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Model  Verification,  Validation,  and  Accreditation  (VV&A) 
Common  Ground  Within  the  M&S  Community 


For  the  past  two  years,  the  Depart¬ 
ment  of  Defense  has  placed 
increased  emphasis  on  both 
assessing  and  enhancing  the  credibility  of 
its  models  and  simulations  (M&S) 
through  the  development  of  practical, 
cost  effective  VV&A  methods  and  tech¬ 
niques,  as  well  as  seeking  the  tools  need¬ 
ed  to  automate  these  efforts.  This  work  is 
designed  to  be  extensible  across  the  spec¬ 
trum  of  DoD  M&S  that  support  various 
analytic,  training,  wargaming/exercise, 
and  acquisition  activities.  These  VV&A 
methodologies  must  address  problems 
and  issues  currently  facing  our  DoD  prac¬ 
titioners  such  as: 

•  Data  availabilty,  quality,  and  standard¬ 
ization; 

•  V&V  sufficiency  (How  much  V&V  is 
enough?); 

•  Appropriate  V&V  methods  for  stand¬ 
alone  or  distributed  model  suites  (or 
federations)  used  for  specific  domain 
applications; 

•  ,  Appropriate  method/level  of  V&V  for 
the  large  number  of  existing  (legacy) 
systems  that  will  be  used  until  their 
successor(s)  becomes  available;  and, 
most  importantly 

•  What  are  the  costs — and  benefits — 
realized  from  VV&A? 

The  Defense  Modeling  and  Simulation 
Office  (DMSO)-sponsored  VV&A  Tech¬ 
nical  Working  Group  (TWG)  serves  as 
the  DoD  forum  which  identifies  these 
VV&A  problem  areas  and  draws  together 
the  appropriate  expertise  to  determine 
possible  approaches  to  these  issues. 
Many  of  these  issues  are  breaking  new 
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ground  within  DoD;  however,  have  these 
issues  been  addressed  by  M&S  organiza¬ 
tions  outside  of  DoD?  The  TWG  mem¬ 
bers  felt  that  a  conference  comparing  the 
VV&A  challenges,  methods,  tools,  and 
results  experienced  by  the  various  M&S 
organizations  would  bring  fresh  perspec¬ 
tives  to  the  DoD  effort.  Where  do  we 
share  common  ground  of  what  VV&A  is 
or  should  be?  Where  do  our  experiences 
diverge — and  why?  More  importantly, 
what  methods  and  tools  are  found  to  be 
most  cost-effective  for  particular  model¬ 
ing  applications? 

Subsequently,  a  Colloquium  involving 
M&S  experts  from  government,  industry, 
and  academia  was  conducted  at  Fort 
Belvoir,  VA  on  28  and  29  Sep  95.  The 
agenda  consisted  of  a  series  of  presenta¬ 
tions  by  practioners  who  are  actively 
engaged  in  their  organizations’  V&V 
efforts.  Two  presentations  specifically 
addressed  software  V&V  activities;  the 
remaining  presentations  were  case  studies 
illustrating  how  different  organizations 
approached  model  V&V.  (An  Executive 
Summary  and  briefing  slides  for  the  each 
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presentation,  as  well  as  an  expanded 
white  paper,  is  available  for  download 
from  the  DMSO  Home  Page  — 
http://www.dmso.  mil.).  These  presenta¬ 
tions  included: 

•  Mr.  George  Neat,  U.S.  Department  of 
Transportation,  Crack  Failure  Models 
for  Aircraft; 

•  Dr.  Peter  Knepell  and  Dr.  Thomas 
Curry,  Logicon  RDA,  VV&A  for  an 
Acquired  Immune  Deficiency  Syn¬ 
drome  (AIDS)  Model; 

•  Mr.  John  Hinkle,  NASA  Software 
IV&V  Facility,  IV&V  of  the  Interna¬ 
tional  Space  Station  (Software); 

•  Mr  Brian  Keltch,  National  Institute  for 
Petroleum  and  Energy  Research, 
Department  of  Energy's  Tertiary  Oil 
Recovery  Information  System  VV&A 
Issues; 

•  Mr.  Bruce  Hicks,  NOAA  Air 
Resources  Laboratory,  (Atmospheric) 
Dispersion  Modeling; 

•  Dr.  Lance  Miller,  SAIC,  Quality  Soft¬ 
ware:  Lessons  Learned 

•  Mr.  Jeff  Gordon,  U.S.  Department  of 
Transportation,  Lessons  Learned  Using 
Finite  Element  and  Non-Linear  Opti¬ 
mization  Models  to  Predict  Railroad 
Component  Stress  and  Performance  in 
Service;  and 

•  Dr.  Michael  Kleinherger,  National 
Highway  Traffic  Safety  Administra¬ 
tion,  Development  and  Validation  of 
Human  Anatomic  Finite  Element  Mod¬ 
els; 

These  presentations  served  as  both  a 
(See  VV&A  p,  30) 
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MORS  1996-97  a  Year  for  ‘‘Reestablishing  the  Foundations  of  Analysis” 


In  the  June  issue  of 
PHALANX  you 
read  the  “Plat¬ 
form”  articles  prepared 
by  each  of  the  Candi¬ 
dates  running  to  be 
your  new  MORS  Pres¬ 
ident.  We  had  each 
prepared  very  logically 
similar  articles  in  iso¬ 
lation  that  were  each 
centered  on  the  theme 
of  Challenge,  Change  and  Opportunity.  Not 
having  seen  the  articles  prepared  by  Priscil¬ 
la  Glasow  and  Jackie  Henningsen,  I  was 
very  surprised  upon  picking  up  my  copy  of 
the  PHALANX  to  find  that  even  our  titles 
were  very  similar.  In  keeping  with  these 
thoughts  and  my  platform  statements,  this 
will  be  a  MORS  year  of  leveraging  technol¬ 
ogy,  managing  and  controlling  change  and 
preparing  our  Society  and  Profession  for  the 
next  (some  would  even  say  current)  era  of 
Joint,  Interoperability  and  Coalition  Opera¬ 
tions.  Our  community  must  be  prepared  to 
accommodate  the  explosive  growth  in  infor¬ 
mation/communication  technologies  and 
take  advantage  of  that  opportunity  for  creat¬ 
ing  the  proper,  up  to  date  tools  for  our  analy¬ 
sis  profession  with  applications  in  this  new 
environment. 

Your  new  slate  of  elected  Society  Offi¬ 
cers  also  includes  Jerry  Kotchka,  VP 
(Finance  and  Management);  Kerry  Kelley, 
VP  (Meeting  Operations);  Denny  Baer, 
VP(Professional  Affairs);  and  Joe  Tatman, 
Secretary.  This  is  an  incredibly  capable  and 
experienced  team  that  has  spent  many  years 
in  MORS  positions  even  before  their  present 
term  on  the  Board  of  Directors.  In  our  for¬ 
mative,  planning  meetings  at  Fort  Leaven¬ 
worth  we  came  to  rapid  agreement  on  the 
need  for  wider  participation  and  empower¬ 
ment  of  our  membership  at  large  and  will  be 
soliciting  not  only  your  continuing,  but 
increasing  levels  of  support  and  activity  for 
the  Society.  We  have  been  involved  in 
adding  Non-Board  Members  to  each  stand¬ 
ing  MORS  committee  and  are  always  look¬ 
ing  for  even  more  volunteers.  We  will  be 
soliciting  your  help  very  soon  in  our  effort 
to  “reengineer”  the  Working  Group  /  Com¬ 
posite  Group  Structure.  A  special  commit¬ 
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tee  was  formed  under  Kerry  Kelley,  chaired 
by  Sue  Iwanski  which  will  be  laying  out 
their  plan  of  action  to  accomplish  the  details 
this  year  for  implementation  at  the  66^^ 
MORSS  (the  MORSS  after  next)  in  1998. 

The  above  MORS  Theme  for  this  year  is 
a  consensus  product  of  your  officer  team. 
As  we  labored  last  month  over  the  articula¬ 
tion  of  the  Society’s  thrust  for  the  coming 
year,  these  simple  words,  in  our  mind, 
expressed  what  we  are  about  in  the  Society 
for  the  next  year. 

Looking  Back 

Our  immediate  Past  President,  Chris 
Fossett  has  led  the  Society  in  a  very  warm 
and  efficient  manner  over  the  past  year  and 
presents  me  with  a  real  challenge  to  keep  up 
with  her  effective  style  of  leadership.  I’m 
most  thankful  for  the  excellent  shape  in 
which  she  has  handed  over  your  Society  to 
our  new  Officer  Team  -  and  that  she  and 
our  Advisory  Directors  (AD)  are  available 
to  guide  our  steps  during  the  coming 
months.  The  slate  of  AD’s  who  very  gra¬ 
ciously  agreed  to  continue  their  active 
involvement  with  the  Board  of  Directors 
represents  a  dedication  to  MORS  and  our 
professional  community  that  makes  this 
organization  so  very  special. 

The  64^  MORSS  is  now  history!  What 
a  great  Symposium!!  We  have  much  to  be 
proud  of  in  the  success  of  our  64^^  at  Fort 
Leavenworth.  Our  Program  Chair,  Dick 
Helmuth  and  his  team  pulled  off  a  first  class 
Symposium  complete  with  some  of  the  best 
site  coordination  work  that  we  have  ever 
seen! !  A  big  special  “Thank  You”  to  Dick, 
Mike  Bauman,  Phil  Kubler,  Annie  Pate- 
naude,  Stuart  Starr,  Kerry  Kelley  and  the 
entire  Program  Committee  for  a  job  very 
well  done.  We  are  all  most  appreciative  for 
a  professional  job  that  produced  a  “warm” 
Kansas  atmosphere  for  both  a  productive 
and  most  enjoyable  Symposium.  Take  the 
time  to  tell  the  64^^  Team  to  include  our 
MORS  Staff  how  much  we  appreciate  their 
hard  work.  Dick  Wiles  tells  me  that  the 
final  number  stands  at  920  which  ranks  as 
the  4th  MORSS  in  terms  of  attendance.  Not 
bad  for  Kansas  in  August  -  er  June! 

Wanner  Award:  Congratulations  again 


to  our  MORS  friend.  Fellow,  Past  President 
and  Chair  of  numerous  MORS  Meetings, 
Ed  Brady  on  his  selection  for  the  Wanner 
Award.  Ed  becomes  the  19th  Wanner 
Award  Laureate! 

Looking  Forward 

After  collectively  basking  in  the  glow  of 
our  great  past  year  and  Symposium,  it  is 
time  to  roll  up  our  sleeves  and  start  to  work 
on  the  next  great  MORS  year.  We  have  a 
tremendous  set  of  Vice  Presidents  and  Com¬ 
mittee  Chairs  along  with  a  large  slate  of 
Non-Board  Members  who  have  volunteered 
to  help  with  our  Committees. 

Congratulations  to  your  following  new 
members  of  the  Board  of  Directors  which 
were  elected  to  their  standard  four  year  term 
at  our  June  Meeting:  Col  Tom  Allen,  Air 
Force  Studies  and  Analysis  Agency;  Dr  A1 
Brandstein,  Marine  Corps  Combat  Devel¬ 
opment  Command;  RADM  Pierce 
Johnson,  Naval  Reserve  Readiness  Com¬ 
mand,  Region  6;  MAJ  Willie  McFadden, 
Army,  HQ  TRADOC;  Dr  Julian  Palmore, 
University  of  Illinois,  and  Editor  of  PHA¬ 
LANX,  and  Dr  Roy  Rice,  Teledyne  Brown 
Engineering.  We  are  proud  to  have  such 
accomplished  and  widely  diversified  Mor- 
sians  joining  the  Board  of  Directors  and 
expect  great  things  out  of  them  over  their 
next  years  of  service.  This  year  there  were 
18  outstanding  nominations  voted  on  to  fill 
the  above  six  slots. 

The  65th  MORSS  is  off  to  a  good  start 
with  Harry  Thie,  Program  Chair  conduct¬ 
ing  kick  off  meetings  with  his  excellent  Pro¬ 
gram  Committee  at  Fort  Leavenworth.  He 
is  out  in  front  with  a  superb  committee  that 
is  deep  in  talent  and  long  on  reputations  for 
hard  work.  It  is  most  impressive  to  see  the 
degree  of  organization  that  is  already  in 
place  for  next  summer.  Let’s  keep  at  it!! 
Harry  has  put  a  detailed  “Planning  Sched¬ 
ule”  in  place  that  has  some  very  important 
work  to  be  done  over  the  next  few  months. 
Please  be  thinking  early  about  the  65th 
MORSS  theme  of  “Analysis  for  Complex 
and  Uncertain  Times”.  We  need  to  incorpo¬ 
rate  that  thought  into  everything  we  do  this 
year.  This  theme  also  fits  very  well  with  the 

(See  MORS  PRESmENT  p.  29} 
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Recognizing  Excellence 


Our  members 
are  making 
important 
contributions  to 
strengthening  nation¬ 
al  security.  We  do 
this  by  providing 
information  that 
improves  defense 
decision  making.  I 
hope  you  will  join  me 
in  congratulating  a 
few  of  our  fellow  members  who  have 
recently  been  recognized  for  outstanding 
accomplishments  as  researchers  and  stu¬ 
dents  of  operations  research. 

Koopman  Prize 

At  the  Washington,  DC  INFORMS 
meeting,  I  was  pleased  to  watch  Professor 
Keith  Womer,  MAS’s  Prize  Committee 
Chair,  award  the  Koopman  Prize  for  the 
outstanding  military  operations  research 
study  published  in  1994  to  Dr.  Hanif  D. 
Sherali,  Virginia  Polytechnic  Institute 
and  State  University,  Dr.  Youngho  Lee, 
U.  S.  West  Advanced  Technologies,  and 
Dr.  Donald  D.  Boyer,  Naval  Surface 
Warfare  Center.  Their  winning  study, 
“Scheduling  Target  Illuminators  in  Naval 
Battle-Group  Anti-Air  Warfare”  was  pub¬ 
lished  in  Naval  Research  Logistics.  This 
outstanding  effort  was  judged  the  superior 
piece  among  an  unusually  strong  field  of 
candidate  papers.  The  papers  were  evalu¬ 
ated  by  the  Koopman  Prize  Committee 
consisting  of  Professor  Jeff  Camm,  Uni¬ 
versity  of  Cincinnati,  Dr.  Bob  Tripp, 
Synergy,  Inc.,  and  Professor  Don  Barr, 
U.  S.  Military  Academy. 

The  Koopman  Prize  is  named  for  the 
late  Bernard  Koopman  (1900-1981),  a 
pioneer  in  the  field  of  operations  research 
who  was  active  in  the  founding  of  the 
Operations  Research  Society  of  America 
(ORSA)  and  served  as  its  president  in 
1956.  Koopman  served  as  an  operations 
research  liaison  between  the  U,  S.  Depart¬ 
ment  of  Defense,  the  U.  K.  military  estab¬ 
lishment  and  NATO,  and  played  a  key 
role  in  introducing  operations  research  as 
a  permanent  NATO  activity. 


Awards  for  Excellence  in  Opera¬ 
tions  Research 

One  of  MAS’  most  important  missions 
is  to  encourage  students  of  military  opera¬ 
tions  research.  We  do  that  by  fostering 
Student  Chapters  at  military  academies 
and  universities,  providing  speakers, 
encouraging  participation  in  INFORMS 
activities,  and  most  importantly  for 
today’s  message,  by  presenting  awards  to 
the  outstanding  operations  research  stu¬ 
dents  at  the  military  academies. 


I  was  proud  and  honored  to  represent 
MAS  at  graduation  ceremonies  at  the 
Naval  Academy  and  the  Air  Force  Acade¬ 
my.  In  Colorado  Springs,  I  presented  a 
MAS  prize  plaque  and  bronze  eagle  to  Air 
Force  Cadet  Andrew  E.  Coop  who  was 
unanimously  selected  by  faculty  represen¬ 
tatives  from  four  Academic  Departments. 
Cadet  Coop  was  number  one  academical¬ 
ly  in  his  class  in  the  Operations  Research 
major  and  was  also  named  Outstanding 
Squadron  Commander,  He  was  president 
of  the  Omega  Rho  Operations  Research 
Honor  Society  at  the  Air  Force  Academy. 

I  attended  a  graduation  award  ceremo¬ 
ny  in  Annapolis  where  I  presented  a  MAS 
prize  plaque  and  a  check  for  $250  to  Mid¬ 
shipman  Terrence  Nawara  who  excelled 
in  courses  in  operations  research.  His 
selection  was  also  based  on  a  senior  pro¬ 
ject,  titled  “Problems  with  Multi-line 
Sonar  Detection,”  that  was  initiated  dur¬ 
ing  an  internship  at  the  Institute  for 
Defense  Analyses.  Midshipman  Nawara 


Dr.  Stephen  J. 
Balut 

MAS  President 


was  selected  for  the  nuclear  power,  sub¬ 
surface  warfare  specialty. 

Cadet  Tyler  B.  Smith  was  awarded  a 
MAS  prize  plaque  and  $250  worth  of 
books  at  the  Awards  Convocation  for  the 
graduating  class  of  1996  at  West  Point. 
Cadet  Smith  earned  the  highest  GPA  in 
10  core  operations  research  courses.  The 
award  was  presented  by  Colonel  David 
C.  Arney,  Head  of  the  Mathematical  Sci¬ 
ences  Department,  U,  S.  Military  Acade¬ 
my.  I  want  to  thank  Professor  Don  Barr 
for  assisting  with  the  arrangements  for  the 
award  of  this  prize. 

Other  Nevrs 

In  other  Society  news,  I  officially 
appointed  the  Nominating  Committee  for 
our  upcoming  election  of  officers.  The 
Chair  of  the  Committee  is  our  President¬ 
elect,  Professor  Tom  Gulledge.  He  is 
accompanied  on  the  Committee  by  Dr. 
Dean  Hartley  and  Dr.  Peter  Cherry,  our 
two  most  recent  past  MAS  Chairs,  Watch 
for  the  announcement  of  nominees  in  this 
column. 

Over  the  past  few  weeks,  the 
INFORMS  Business  Office  discovered 
that  a  large  portion  of  MAS  members 
were  not  receiving  PHALANX.  The  root 
cause  of  this  has  been  identified  and  is 
related  to  shifts  in  administrative  respon¬ 
sibilities  that  followed  the  merger  of 
ORSA  and  TIMS.  A  procedure  has  been 
reestablished  and  implemented  that 
ensures  that  all  members  will  receive  each 
issue.  To  those  of  you  who  missed  back 
issues,  we  are  very  sorry  for  this  and  we 
will  be  in  touch  with  you. 

I  am  pleased  to  report  that  membership 
in  MAS  is  increasing;  Clusters  planned 
for  Atlanta  and  San  Diego  look  great;  we 
are  rewriting  our  Bylaws  to  reflect  Soci¬ 
ety  status;  and  we’re  negotiating  to  make 
it  easier  and  cheaper  for  you  to  subscribe 
to  our  journal.  Military  Operations 
Research.  Its  a  good  time  to  get  more 
actively  involved,  O 
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VEEPS  PEEP 


“Meetings:  What’s  Ahead” 


^  nhance  the 
1^  quality  and 
lb  ^usefulness 
of  classified  and 
unclassified  military 
operations  research”  - 
the  stated  purpose  of 
MORS  serves  to 
underscore  the  goals 
and  objectives  of  the 
meetings  sponsored 
by  our  society.  In 
reflecting  upon  our  society’s  most  recent 
meetings  and  in  looking  ahead  to  plans  for 
the  coming  year  it  is  useful  to  check  our 
accomplishments  and  goals  with  the  broad¬ 
ly  stated  purpose  of  our  MORS.  In  doing 
so,  I  believe  that  you  will  find,  as  I  did,  a 
myriad  of  activities  within  meeting  opera¬ 
tions  which  help  further  this  purpose. 
During  the  next  year,  I  am  charged  with 
the  Meeting  Operations  business  of 
MORS.  As  the  meeting  operations  respon¬ 
sibilities  have  transitioned  to  me,  I  find  that 
I  have  been  given  an  extremely  orderly 
state  of  affairs  -  thanks  to  the  strong  lead¬ 
ership  of  Fred  Hartman,  who  was  VPMO 
last  year  and  our  President  this  year,  Chris 
Fossett,  our  Past  President,  and  the  entire 
MORS  staff.  In  the  next  few  paragraphs  I 
hope  to  update  you  on  recent  accomplish¬ 
ments  and  inform  you  as  to  what  to  expect 
in  the  coming  year. 

64th  MORSS 

The  64^^  MORSS  in  Ft.  Leavenworth 
was  an  overwhelming  success.  We  owe 
many  thanks  to  the  Program  Chair,  Dick 
Helmuth,  and  to  his  entire  Program  Com¬ 
mittee,  with  special  thanks  to  Mike  Bau¬ 
man,  Phil  Kubler,  Annie  Patenaude  and 
Stuart  Starr.  The  latest  tally  on  atten¬ 
dance  indicated  that  we  were  920  members 
strong  -  the  4^  largest  MORSS  attendance 
figure.  I  attribute  this  mostly  to  the  out¬ 
standing  program  but  I  also  cannot  over¬ 
look  the  fine  location  as  well  (being  a  mid- 
westerner  myself).  I  hope  all  who 
participated  found  the  symposium  useful 
and  are  actively  involved  in  leveraging 
technology  in  their  military  analyses. 


65th  MORSS 

The  65th  MORSS  will  be  at  the  Marine 
Corps  University  in  Quantico,  VA  on  10- 
12  June  1997.  The  theme  will  be  “Analy¬ 
sis  for  Complex,  Uncertain  Times”.  Dr. 
Harry  Thie,  the  Program  Chair,  is  already 
working  diligently  at  the  preparations  for 
this  meeting  and  has  also  written  an  article 
in  this  edition  of  the  PHALANX.  Please 
see  his  article  for  additional  information 
and  for  the  points  of  contact  on  his  65^ 
MORSS  committee.  Jay  Wilmeth,  the 
Working  Group/Composite  Group  Coordi¬ 
nator  and  a  member  of  the  65^  MORSS 
committee  will  be  working  with  the 
WG/CG  Chairs,  Co-chairs  and  Advisors  to 
ensure  a  challenging  agenda  for  all. 

66th  MORSS 

Believe  it  or  not,  plans  are  already  well 
underway  for  the  66^  MORSS.  The  66^ 
MORSS  will  be  hosted  by  the  Naval  Post¬ 
graduate  School  in  Monterey,  CA.  We  are 
fortunate  to  have  RADM  Pierce  Johnson 
as  the  Program  Chair  for  the  66^  MORSS. 
With  Pierce  Johnson  at  the  helm  and  the 
very  popular  NPS  location,  the  66^^ 
MORSS  promises  to  be  yet  another  out¬ 
standing  conference. 

Special  Meetings 

Dr.  Stuart  Starr  has  once  again  agreed 
to  take  the  lead  as  the  Chair  of  the  Special 
Meetings  Committee.  He  served  MORS 
excellently  in  this  capacity  last  year  and 
under  his  leadership  we  benefited  from 
several  meetings: 

•  Developing  a  Framework  for  Joint 
Mobility  Analysis  (26-28  Sep  95) 

•  Advanced  Distributed  Simulations  for 
Analysis  1996  (ADSA-1996)  (30  Jan-1 
Feb  96). 

The  workshop  on  joint  mobility  analy¬ 
sis  was  chaired  by  Mr.  Jim  Johnson, 
OASD(PAE)  and  advanced  military  OR 
through  development  of  a  framework  for 
the  analysis  of  mobility  issues.  The  focus 
of  ADSA-1996  was  on  the  appropriate  use 
of  ADS,  identification  of  advantages,  limi¬ 


tations  and  areas  for  which  improvements 
offer  greatest  potential,  and  suggested 
ways  to  analyze  and  test  the  quality  of 
ADS-based  tools.  The  workshop  was 
chaired  by  MORS  Fellow  and  Wanner 
Award  Laureate,  Ed  Brady  —  an  article 
appears  in  this  issue  of  the  PHALANX, 
page  13. 

Stuart  Starr  has  preparations  well  in 
hand  for  a  mini-symposium  on  1-3  Octo¬ 
ber  1996  in  McLean,  VA  entitled  “First 
Order  Analysis:  Exploring  Quick 
Response  Analysis  Requirements  and 
Methodologies  (QRAM).  Dr.  Jackie  Hen- 
ningsen  is  the  General  Chair;  the  Techni¬ 
cal  Chair  is  Doug  Williams  and  Co-chairs 
are  Dr.  Roy  Rice  and  Bob  Statz.  The 
meeting  will  provide  a  focus  on  existing 
methodologies  that  allow  analysts  to  gain 
first  order  understanding  of  problems  prior 
to  allocation  of  analytic  resources  and  to 
provide  quick  response  results  for  senior 
DoD  leaders. 

In  the  planning  stages  are  preparations 
for  two  additional  special  meetings: 

•  “State  of  the  Art  in  Warfare  Analysis” 
(late  1996  -  Chairs:  Gene  Visco,  FS  and 
Dr.  A1  Brandstein) 

•  “Modeling  and  Simulation  (M&S)  Link¬ 
age”  (Spring  1997  -  Chairs:  Dr.  Marion 
Williams,  FS  and  Jim  Sikora,  FS ) 

The  objective  of  the  meeting  on  “State 
of  the  Art  in  Warfare  Analysis”  is  to  build 
on  a  recent  international  workshop  on  ana¬ 
lytic  approaches  for  the  study  of  future 
conflict.  The  “M&S  Linkage”  mini-sym¬ 
posium  is  intended  to  address  the  state-of- 
the-art  in  M&S  linkage  and  current  issues 
and  concerns  for  this  exploding  trend. 
Look  for  additional  information  in  upcom¬ 
ing  PHALANX  issues. 

One  final  item  which  we  are  exploring 
is  the  possibility  for  MORS  to  organize  an 
International  Symposium  for  selected 
allies’  participation.  Our  research  is  only 
in  its  infancy  at  this  point  but  the  concept 
was  suggested  by  the  MORS  Sponsors  and 
Past  Presidents  and  we  will  investigate  this 
very  exciting  potential! 

VEEPS  PEEP  p.  35 
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Objects,  Objections,  and  Models 
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Introduction 

In  a  previous  arti¬ 
cle  in  PHALANX, 
I  took  an  abbrevi¬ 
ated  shot  at  the  current 
enthusiasm  for  huge, 
objected-oriented,  dis¬ 
tributed  interactive 
simulations  (“Measur¬ 
ing  Pants  Legs,” 
December  1995). 
Both  my  current 
research  and  the 
recent  lead  article  in 
PHALANX  (“Simulation  for  C^ISR by 
Chuck  Marshall  and  Dr.  Randy  Garrett, 
March  1996)  now  permit  me  to  attack  on  a 
broader  front  about  the  viability  of  these 
fashions  in  analysis.  Fm  not  acquainted 
with  Dr.  Garrett,  but  Chuck  Marshall  is  a 
former  colleague  of  mine  and  I  fully  expect 
him  to  respond  (read  shoot  back). 

For  the  last  few  years,  Fve  been  helping 
develop  an  optimization  model  for  muni¬ 
tions  planning  and  campaign  analysis,  and 
my  patrons  recently  asked  me  to  add 
C^ISR  to  this  model  in  some  form.  Mar¬ 
shall  and  Garrett’s  article  appeared  at  about 
the  same  time,  so  Fd  hoped  it  would  pro¬ 
vide  some  insight  as  to  what’s  going  on  in 
the  mainstream  with  C^ISR.  Marshall  and 
Garrett’s  piece  did  give  me  a  great  deal  of 
insight,  but  not  the  type  I  was  seeking.  As 
a  result,  I  offer  this  short  piece  as  both  a 
response  to  their  claims  and  a  reminder  to 
the  community  that  fashionable  techniques 
and  useful  analyses  are  not  the  same. 


Their  Objects  and  My  Objections 

Marshall  and  Garrett  are  clearly  enam¬ 
ored  of  objects.  They  call  for  a  new  class 
of  simulations  to  be  built  entirely  at  the 
entity  level,  and  even  suggest  using  objects 
to  represent  things  such  as  tactics  and  doc¬ 
trine.  Such  an  approach,  they  assert,  will 
make  life  better,  because  current  aggregate 
models  rely  on  statistical  modeling  of  col¬ 
lections  of  objects,  and  this  clearly  can’t  be 
as  good  as  modeling  the  objects  them¬ 
selves.  In  addition,  they  claim  an  object 
approach  is  more  computable,  because  “the 


computational  load  can  be  distributed 
among  multiple  processors,  located  at  vari¬ 
ous  sites.” 

Marshall  and  Garrett  reveal  a  lot  in  their 
account  for  why  rtSR  is  hard  to  model: 
the  primary  reason,  they  say,  has  been  the 
lack  of  computational  power,  while 
achieving  representation  of  human  decision 
making  and  communications  has  been  sec¬ 
ondary.  This  proposition  is  amazing 
enough  that  I  will  repeat  it:  our  biggest 
problem  modeling  C^ISR  is  the  lack  of 
computational  capability;  the  fact  that  we 
don’t  know  how  to  model  the  essential  ele¬ 
ments  and  their  interactions  isn’t  that  big  a 
deal.  The  authors  are  convinced  that 
advances  in  computing  and  the  switch  to 
object  orientation  will  overwhelm  these 
secondary  problems,  and  somehow  solve 
the  problems  of  modeling  C^ISR.  I  con¬ 
tend  this  argument  is  wrong,  and  is  central 
to  a  larger  debate  about  object-oriented 
methods. 

Under  the  subheading  “Modeling  Infor¬ 
mation  as  an  Object,”  the  authors  claim  that 
“[traditionally]  in  order  to  accommodate 
the  contribution  of  intelligence,  stochastic 
relationships  were  developed  between 
objects.  As  a  result,  the  value  of  intelli¬ 
gence  is  difficult  to  ascertain  and  its  rela¬ 
tionships  and  other  factors  on  the  battlefield 
are  never  determined.”  It  appears  that  this 
statement  points  to  a  collection  of  bad  mod¬ 
els  and  rejects  analytical  approaches  based 
on  those  models’  behavior.  Marshall  and 
Garrett  would  no  doubt  include  the  opti¬ 
mization  Fm  working  on,  as  it  has  no 
C^ISR  whatsoever  (except  for  rudimentary 
BDA),  and  no  objects  of  any  kind.  Now, 
this  model’s  current  shortcomings  include 
not  addressing  G^ISR,  but  to  say  these 
shortcomings  are  due  to  its  lack  of  “real, 
tangible,  and  visible”  C^ISR  objects  is 
insupportable:  again,  C^ISR  is  simply  not 
modeled.  Nonetheless,  Marshall  and  Gar¬ 
rett  make  just  this  argument  when  they  say 
later  that  “the  current  structure  of  most 
models  does  not  lend  itself  to  illuminating 
the  worth  of  information.”  They  haven’t 
proved  that  analytical  methods  are  inca¬ 
pable  of  representing  G^ISR;  they  have 
only  stated  that  some  analytical  models  do 


C^ISR  badly.  Does  that  say  anything  con¬ 
clusive  about  what  analytical  approaches 
could  do  if  someone  bothered  to  invest  in 
them? 

Consider  two  phenomena  that  are  cur¬ 
rently  modeled  using  analytical  methods: 
first,  the  movement  of  hurricanes;  and  sec¬ 
ond,  the  inflation  factors  used  by  the  gov¬ 
ernment  for  various  policy  decisions.  Both 
are  tremendously  complex  and  governed  by 
processes  that  we  don’t  understand  well 
(like  C^ISR).  However,  we  have  used 
these  models  for  a  long  time  and  enjoyed  a 
fair  amount  of  accuracy.  If  we  attempted  to 
model  either  of  these  phenomena  using  the 
molecular  approach  advocated  by  Marshall 
and  Garrett,  how  long  would  it  take  us  to 
even  develop  a  model?  Don’t  be  fooled  by 
romantic  terms  such  as  object  request  bro¬ 
kers,  inheritance,  and  polymorphism  —  any 
model  developed  under  these  philosophies 
will  be  very  large  and  very  complicated. 

Even  if  I  was  willing  to  accept  object 
modeling  at  the  level  proposed  by  Marshall 
and  Garrett,  I  would  balk  at  the  greatest  gap 
in  their  approach:  the  demotion  of  the 
machine  representation  of  human  decision 
making  to  a  second-string  problem.  We 
know  human  logic  is  extremely  variable 
and  human  choices  and  priorities  are  gener¬ 
ally  not  transitive;  the  AI  community  con¬ 
tinues  to  struggle  with  representing  even 
the  simplest  decision  using  machine  logic. 
However,  to  merely  assure  us  that  “work 
on  higher-level  ‘strategic  thinking’  is  on  the 
way,”  grossly  overstates  our  current  or 
near-term  capabilities.  At  the  most  recent 
INFORMS  meeting  in  Washington,  D.C.,  I 
attended  a  military  applications  seminar 
that  was  discussing  AI  approaches  to  com¬ 
mand  in  models.  One  speaker  was  cover¬ 
ing  the  capabilities  of  his  organization’s 
ground  combat  model,  which  contained  a 
very  complex  Al-based  command  system. 
At  the  end,  I  asked  what  it  took  to  build  a 
rule  set  for  this  model.  The  speaker  replied 
he  thought  it  would  take  three  Army  offi¬ 
cers  with  both  command  experience  and  an 
MS  or  Ph.D.  in  artificial  intelligence  about 
a  year  to  do  the  job,  for  a  particular  sce¬ 
nario.  How  responsive  is  this? 
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Objects,  Combinatorics,  and  the 
Lessons  of  Regression 

The  primary  danger  of  using  object-ori¬ 
ented  methods  is  that  they  are  combinatori- 
ally  explosive.  Marshall  and  Garrett,  while 
bemoaning  the  lack  of  explicit  sensor  mod¬ 
eling,  say  that  “sensors  want  to  see  ‘things,’ 
not  an  equation  representing  thousands  of 
things.”  Let’s  consider  the  ramifications  of 
stuffing  thousands  and  thousands  of  these 
“things”  into  a  simulation. 

Suppose  my  combat  entity  has  1000 
objects  in  it.  Furthermore,  I  want  to  endow 
these  objects  with  some  behavior  they  use 
when  some  set  of  sensors  is  looking  for 
them.  Clearly,  this  behavior  should  be  a 
function  of  how  many  of  them  there  are  in 
the  immediate  area,  so  we  have  an  interac¬ 
tion.  Marshall  and  Garrett  tell  us  we  need  to 
be  more  worried  about  terrain,  so  these 
behaviors  should  also  be  a  function  of  the 
terrain  (which  also  probably  interacts  with 
the  number  of  objects  in  a  region).  It’s 
starting  to  look  like  everything  interacts 
with  everything,  which  leaves  me  with 
2 1000  variants  of  my  behaviors  to  describe. 
This  would  satisfy  most  people’s  definition 
of  a  big  number. 

Perhaps  the  object-oriented  modeler  will 
tell  me  that  I  really  don’t  have  2^^^  possi¬ 
ble  interactions  to  worry  about,  as  many  of 
the  higher-order  interactions  have  negligi¬ 
ble  effects  and  this  unit  of  1000  objects  is 
commanded  by  a  commander  object  that 
tells  them  what  to  do  anyway.  Considering 
just  the  environmental  interactions,  that 
leaves  me  with  perhaps  just,  say,  2^^  inter¬ 
actions?  That’s  only  a  million  pieces  of 
data  for  one  unit  being  looked  at  by  one  set 
of  sensors. 

This  is  the  bitter  lesson  of  combina¬ 
torics,  and  it  completely  destroys  Marshall 
and  Garrett’s  notion  that  I  can  somehow 
compute  my  way  out  of  this  box.  For  the 
example  above,  2^^  interactions  are  proba¬ 
bly  the  fewest  interactions  necessary  to  pro¬ 
vide  the  level  of  decomposability  that  an 
object  approach  requires.  This  is  for  one 
unit  and  one  event!  The  point  is,  it  doesn’t 
matter  how  much  computing  power  you 
have  available;  we  should  not  decompose 
things  that  do  not  need  to  be  decomposed, 
because  the  difficulty  of  modeling  internal 
interactions  so  that  they  mimic  the 
observed  aggregate  behavior  swamps  the 
difficulty  of  modeling  aggregate  behavior. 
Consider  a  simple  physical  example.  New¬ 
ton’s  Law  of  Cooling  states  that  the  rate  of 


change  of  the  temperature  of  an  object  is 
proportional  to  the  difference  between  the 
object’s  temperature  and  that  of  the  sur¬ 
rounding  air,  and  as  such  is  modeled  with  a 
first-order  differential  equation.  Now,  most 
physicists  would  agree  that  this  is  a  pretty 
rotten  model  of  what’s  going  on  at  the  mol¬ 
ecular  level,  considering  air  movements, 
non-homogeneity  of  the  materials,  and  so 
on,  but  somehow  this  innocuous  equation 
gives  good  answers  for  most  situations.  If  I 
can  get  the  answer  I  need  using  such  a  sim¬ 
ple  equation,  why  bother  with  the  mole¬ 
cules?  This  leads  me  to  my  next  point, 
concerning  the  regression  paradox. 

Most  PHALANX  readers  have  taken  a 
regression  course  sometime  during  their 
careers.  A  clever  statistician  teaching 
regression  will  reveal  to  his  students  that 
R  ,  the  coefficient  of  determination,  can 
never  go  down  if  you  add  more  predictors. 
Aha,  say  the  students.  If  I  can  explain  more 
of  the  total  variability  with  more  predictors, 
I  should  add  a  gazillion  predictors  and  drive 

to  1 .  This  will  get  me  an  A!  Of  course, 
the  students  (and  all  of  us)  discover  the 
awful  truth  sooner  or  later:  not  only  is 
wantonly  adding  predictors  not  a  good 
thing,  it  can  severely  damage  the  predictive 
powers  of  the  model.  As  a  result,  the  best 
regression  model  is  usually  the  parsimo¬ 
nious  one,  and  not  usually  the  one  with  the 
most  predictors. 

Besides,  regressions  of  most  phenomena 
reveal  that  there  are  relatively  small  sets  of 
predictors  that  explain  most  of  variability  in 
the  response.  Why  is  this?  Because  a 
small  set  of  predictors  either  represents  the 
effects  of  a  multitude  of  other  predictors,  or 
the  other  possible  predictors  depend  on  the 
behavior  of  the  important  predictors.  For 
all  of  Marshall  and  Garrett’s  talk  about  the 
need  to  model  at  the  molecular  level,  the 
fact  is  that  few  of  the  molecules  act  inde¬ 
pendently. 

Another  reality  intrudes  on  this  party. 
Given  that  collecting  data  for  these  models 
costs  a  lot  of  time  and  a  lot  of  money,  I 
don’t  want  to  maintain  information  on 
something  that  has  a  negligible  effect.  In 
every  single  project  I  have  worked  on  for 
the  last  16  years,  data  collection,  investiga¬ 
tion,  recollection,  and  reduction  have  con¬ 
sumed  the  vast  majority  of  the  time 
expended.  However,  Marshall  and  Garrett 
have  nothing  whatsoever  to  say  about  this 
hidden  cost  of  object  models,  and  this  cost 
is  significant. 

Finally,  Marshall  and  Garrett  speak  dis¬ 


paragingly  of  equations  “representing  thou¬ 
sands  of  things,”  and  reject  stochastic 
approaches  to  modeling  information  trans¬ 
mission.  But,  does  anyone  model  without 
using  mathematical  relationships?  For  all 
the  talk  about  object  decomposition,  I  fre¬ 
quently  find  object  modelers  punting  to  the 
dreaded  equations  I  would  expect  them  to 
reject.  At  a  recent  MORS  meeting,  I  heard 
a  speaker  extolling  the  virtues  of  his 
tremendously  detailed  and  complex  combat 
model,  a  model  that  represented  platoons, 
squads,  vehicles,  and  even  individual  build¬ 
ings.  A  lone  questioner  asked,  “How  do 
the  engagements  get  arbitrated?”  After  a 
short  pause,  the  answer  came  back:  “Lan- 
chester  equations.” 

Axioms:  Theirs  and  Mine 

A  useful  way  to  summarize  my  point  is 
to  note  that  distributed-interactive-object- 
oriented-plug-and-play  literature  seems  to 
rely  on  a  small  number  of  axioms.  I  can 
only  infer  these  axioms,  but  they  appear  to 
be  as  follows. 

Object  Axiom  1 :  No  mathematical  func¬ 
tion  relating  the  behaviors  of  multiple 
objects  can  be  as  good  as  modeling  the 
objects  themselves. 

Object  Axiom  2:  Accuracy  and  explain- 
ability  increase  with  the  number  of 
objects. 

Object  Axiom  3:  Data  will  be  available 
for  any  object  decomposition,  no  matter 
how  detailed. 

Clearly,  I  disagree.  I  offer  the  following 
counter-axioms  based  on  my  experience. 
Counter- Axiom  1:  There  exists  a  mathe¬ 
matical  relation  among  objects  that  can 
be  refined  to  accuracy  in  a  Onite  and 
small  number  of  steps,  given  the  right 
team  of  operators  and  analysts  is  calibrat¬ 
ing  it 

Counter-Axiom  2:  Modeling  difficulty 
increases  exponentially  and  explainabili- 
ty  goes  to  0  as  the  number  of  objects 
increases. 

Counter- Axiom  3:  Data  are  never  imme¬ 
diately  available  for  anything,  and  their 
procurement  consumes  the  majority  of 
the  available  study  time  with  probability 
L 

The  risks  inherent  in  using  object-orient¬ 
ed  methods  seem  clear  in  the  face  of  Mar¬ 
shall  and  Garrett’s  overriding  assumption 
(that  we  need  lots  of  objects  to  capture  the 

(See  NUMBERS  FROM  COMBAT  p,  27) 
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Judgments  in  Military  Research: 

The  Elnsive  Internal 


Judgments  are 
used  frequently 
in  military  opera¬ 
tions  research  and 
form  the  basis  for  a 
great  deal  of  military 
policy.  Behind  this 
extensive  use  of  judg¬ 
ments  lies  a  well 
grounded  belief  that 
military  experts  pos¬ 
sess  knowledge  worth  tapping  as  a  reflec¬ 
tion  of  truth  in  their  area  of  expertise,  that 
insight  into  fhture  decisions  can  be  gleaned 
from  decisionmakers’  current  judgments, 
or  that  judgments  are  simply  the  best  or 
only  source  of  useful  information. 
Researchers  obtain  judgments  from  subject 
matter  experts  (SMEs)  to  better  understand 
and  predict  items  of  militaiy  interest  cover¬ 
ing  a  broad  spectrum  of  issues  such  as  the 
location  and  nature  of  future  conflicts, 
required  readiness  levels  and  how  to 
achieve  them,  technology  and  weapon  sys¬ 
tem  selection  and  employment,  force  struc¬ 
ture,  and  so  forth.  Such  judgments  are 
often  analyzed  in  some  fashion  and  their 
implications  for  policy  directly  interpreted. 
Judgments  are  also  used  extensively  to  for¬ 
mulate  combat  simulation  algorithms  and 
thus  they  importantly  influence  simulation 
outcomes  that  serve  as  a  basis  for  policy 
decisions.  Hence,  understanding  the  valid¬ 
ity  of  claims,  inferences,  or  interpretations 
based  on  judgment  data  underlies  their 
credibility  and  usefulness  in  policy  arenas. 

When  it  comes  to  judgments,  the  pre¬ 
dominant  validity  issue  in  the  minds  of 
most  researchers  and  clients  alike  is 
whether  the  judgments  reflect  “real  world” 
behavior,  hereafter  called  external  validity. 
However,  an  experimental  investigation 
into  the  external  validity  aspects  of  judg¬ 
ments  depends  on  having  answered  a  gen¬ 
erally  unknown  internal  validity  question: 
“Does  the  interpretation  of  judgments 
accurately  reflect  the  values  and  thoughts 
of  the  respondents?”  Not  resolving  the 
internal  validity  question  renders  the  exter¬ 
nal  validity  question  moot — external  valid¬ 
ity  cannot  be  appropriately  investigated. 

Because  of  its  dominant  effect  on  the 
credibility  of  research  and  analysis  based 


on  human  judgments  and  the  fact  that  it  is 
commonly  ignored  or  unrecognized,  inter¬ 
nal  validity — what  it  means,  what  it 
affects,  the  penalty  for  ignoring  it,  and  how 
to  achieve  it — is  the  main  subject  of  this 
article.  The  aim  is  to  enlighten  and  encour¬ 
age  researchers  who  choose  to  base  their 
research  on  judgments  to  employ  research 
techniques  that  provide  knowledge  about 
the  internal  validity  of  their  conclusions. 

Validity  Criterion 

To  say  that  a  statement  is  “valid” 
implies  that  it  is  somehow  “true.”  The  cri¬ 
terion  proposed  here  for  concluding  that  a 
statement  is  “true”  is  that  the  statement  has 
been  proposed  as  an  hypothesis  and  tested 
against  and  supported  by  data.  The  notion 
of  using  judgment  data  to  test  hypotheses 
is  new  to  most  military  analysts  and  in  fact, 
hypothesis  testing  is  notably  absent  in  mili¬ 
tary  analysis  when  judgments  form  the 
research  data  base.  This  idea  is  not  new, 
however,  to  measurement  psychologists 
who,  over  the  last  half  century,  have  devel¬ 
oped  experimental  designs  that  make  such 
tests  possible. 

Internal  and  external  validity  issues 
apply  regardless  of  the  type  of  response  or 
format  for  gathering  the  judgment  data — 
interviews,  written  responses,  electronic 
actions — or  the  form  of  the  judgments — 
quantitative,  qualitative,  or  a  written 
record.  To  address  them,  hypotheses  need 
to  be  formulated  and  tested. 

The  SME  Issue 

A  tertiary  “validity”  issue  that  has 
received  a  great  deal  of  attention  in  the 
military  community  is  SME  selection. 
This  issue  is  discussed  briefly  here 
because  it  is  often  given  validity  status  in 
the  military  community  where  the  exter¬ 
nal  validity  question  popularly  focuses  on 
the  qualifications  of  the  respondents  to 
make  correct  judgments.  The  idea  seems 
to  be  that,  if  the  “correct”  SMEs  have 
been  selected,  then  their  statements  are 
valid.  However  resolutions  concerning 
“correct”  SMEs  lie  outside  the  realm  of 
hypothesis  testing  and  need  to  be  distin¬ 
guished  from  issues  that  deal  with  the 
credibility  of  a  researcher’s  interpretation 


Validity  Issue 

of  SMEs’  judgments.  It  is  certainly 
important  to  capture  the  thoughts  and 
ideas  of  the  “right”  people.  But,  who 
they  are  is  not  a  testable  proposition  but  a 
matter  of  definition,  and  a  resolution  lies 
on  one’s  preference  or  selection  criteria, 
not  on  hypothesis  testing. 

Criteria  for  SME  selection  could  be 
based  on  position,  rank,  experience,  rec¬ 
ommendations  by  people  who  should 
know,  selection  by  an  appointed  panel,  test 
scores,  and  so  forth.  Whatever  criteria  are 
used,  some  people  will  probably  not  agree 
with  them  and  question  if  selected  SMEs 
know  what  they  are  talking  about.  Howev¬ 
er  this  opinion,  as  all  opinions,  should  be 
formulated  as  an  hypothesis  and  subse¬ 
quently  tested  rather  than  remain  a  matter 
of  argument.  Such  tests  could  be  conduct¬ 
ed  within  the  hypothesis-testing  frame¬ 
works  described  below. 

Internal  Validity:  Testing 
Hypotheses  About  SMEs’  Values 
and  Thought  Processes 

The  validity  issue  that  is  the  main  focus 
of  this  note  is  referred  to  here  as  internal 
validity.  The  internal  validity  question 
asks  if  researchers’  interpretations  of  judg¬ 
ment  data  accurately  reflect  the  ideas,  val¬ 
ues,  and  thoughts  of  the  SMEs  providing 
the  judgments.  This  question  is  rarely  if 
ever  asked,  let  alone  addressed,  by  the  mil¬ 
itary  analytic  community.  In  fact,  when 
judgments  via  interviews  or  surveys  form 
the  research  data  base,  policy  claims  or 
implications  generally  consist  of  repeating 
or  summarizing  the  contents  of  the  surveys 
or  interviews.  Hypotheses  are  not  tested 
and  thus,  conclusions  cannot  be  wrong.  If 
conclusions  cannot  be  wrong,  one  has  to 
ask  in  what  sense  they  can  be  right. 

The  problem  is  one  of  controlling  vari¬ 
ables  so  that  hypothesis  testing  is  possi¬ 
ble — a  major  experimental  requirement  in 
all  research  aiming  to  draw  causal  conclu¬ 
sions  from  data.  The  basic  idea  in  testing 
hypotheses  about  peoples’  values  and 
thought  processes  is  to  create  questions 
posed  to  respondents  by  simultaneously 
manipulating  two  or  more  factors,  general¬ 
ly  in  factorial  combinations.  Appropriate 
experimental  designs  resulting  from  these 


PHALANX 


8 


September  1996 


factor  manipulations  make  it  possible  to 
test  hypotheses  about  effects  of  factors  on 
judgment  as  well  as  test  among  hypothe¬ 
sized  algebraic  models  to  explain  how 
respondents  valued  and  processed  the  fac¬ 
tor  information.  Respondents’  values 
come  from  the  model  that  explains  the 
data;  they  are  the  model’s  least-squares 
parameter  values  given  the  data.  If  hypothe¬ 
sized  models  and  their  best  fit  paramenter 
values  do  not  account  for  the  judgment 
data  they  are  rejected.  So  in  this  advanced 
psychological  measurement  framework, 
obtaining  respondents’  values  goes  beyond 
the  numbers  or  answers  they  provide  to  the 
researcher;  it  rests  on  determining  an  alge¬ 
braic  measurement  model  that  has  been 
adequately  tested  and  has  received  empiri¬ 
cal  support  for  its  validity. 

An  example  will  serve  to  illustrate  the 
ideas  of  hypothesis  testing  associated  with 
this  measurement  approach.  Suppose  a 
researcher’s  interest  is  to  understand  what 
affects  intelligence  officers’  ability  to  iden¬ 
tify  enemy  maneuver  units  (e.g.,  tank  bat¬ 
talions)  using  information  provided  by 
intelligence  collection  systems.  Assume 
the  three  factors  and  their  factor  levels 
shown  below  are  hypothesized  to  be  key 
factors  affecting  the  situation  assessment 
process  by  which  intelligence  officers 
identify  enemy  units: 

(1)  Coverage:  percent  of  areas  of  interest 
covered  by  the  collection  system;  factor 
levels:  10%,  40%,  70%,  100%; 

(2)  Timeliness:  how  soon  the  information 
is  available  for  use  after  it  is  collected; 
factor  levels:  (5  minutes;  30  minutes; 
90  minutes); 

(3)  Precision:  the  level  of  detail  provided 
about  observed  systems;  factor  levels: 
detection  (something  is  there);  classifi¬ 
cation  (tracked  and  wheeled  systems 
can  be  distinguished);  and  recognition 
(distinction  by  system  type,  e.g.,  tank, 
air  defense,  personnel  carrier). 

Further,  assume  that  the  measure  of  per¬ 
formance  (MOP)  is  the  percent  of  enemy 
units  that  could  be  identified  in  situations 
specified  by  combinations  of  these  factor 
levels.  (These  three  factors  and  MOP 
were  used  in  a  similar  judgment  task 
described  in  Veit  and  Callero^^.)  Intelli¬ 
gence  officers  having  expertise  in  perform¬ 
ing  the  real  task  would  be  asked  to  estimate 
this  MOP  for  each  situation  presented  in  an 
experiment. 

Figure  1  illustrates  some  hypothetical 


judgment  data  that  might  reflect  judgments 
to  36  situations  that  would  have  been  gen¬ 
erated  from  a  4  (Coverage)  x  3  (Timeli¬ 
ness)  X  3  (Precision)  factorial  design.  In 
each  panel  of  Figure  1 ,  judgments  of  the 
percent  of  enemy  units  that  could  be  identi¬ 
fied  is  plotted  on  the  y-axis  as  a  function  of 
Coverage  on  the  x-axis;  a  separate  curve  is 
for  each  level  of  Timeliness,  and  a  separate 
panel  is  for  the  different  levels  of  Preci¬ 
sion. 

The  slopes  of  the  curves  in  Figure  1 
represent  the  effect  of  Coverage  (x-axis)  on 
judgments;  separations  between  the  curves 
represent  the  effect  of  Timeliness,  and  the 
change  in  the  shape  and  position  of  the 
curves  from  panel  to  panel  represents  the 
effect  of  Precision. 

As  can  be  seen,  the  hypothesis  that 
these  factors  affect  judgments  is  supported 
by  the  data.  Also  seen  in  these  hypotheti¬ 
cal  data  are  two  and  three-way  interactions 
among  these  factors  in  their  effects  on 
judgments.  For  example,  in  each  panel  it 
makes  less  of  a  difference  how  timely 
information  is  if  there  isn’t  much  informa¬ 
tion  than  when  more  information  is  avail¬ 
able  (compare  the  vertical  separations 
among  the  three  curves  when  Coverage  is 
10%  on  the  x-axis  with  when  it  is  100%). 
And,  this  interaction  takes  on  a  slightly  dif¬ 
ferent  form  as  curves  rise,  steepen  in  slope, 
and  spread  out  from  Panel  A  to  Panel  C. 
Thus,  these  data  also  support  hypotheses 
about  interactions  among  these  three  vari¬ 
ables.  Tradeoffs  among  the  factors  on 
judgments  can  be  seen  from  the  graphs  of 
the  data.  To  advance  beyond  data  descrip¬ 
tion,  one  can  ask  the  question,  “What  alge¬ 


braic  measurement  model  accounts  for  the 
independent  and  interactive  effects  seen  in 
the  data  and  can  provide  the  values  SMEs 
place  on  these  capabilities?” 

Testing  Among  Algebraic  Models  to 
Measure  Observed  Effects 

With  a  simple  factorial  design  as  shown 
here,  tests  among  algebraic  models  to 
explain  effects  are  limited.  However,  this 
design  serves  to  illustrate  how  models  can 
be  tested  and  rejected  when  wrong.  Let  us 
suppose  that  the  widely  used  expected 
utility  theory  was  proposed  as  a  candidate 
model  to  explain  these  data.  Expected 
utility  theory  is  an  averaging  model  that 
can  be  written: 

Rijk  ^  m[(WcSi  -H  WjSj  -H  WpSj^)/ 
(Wc-HWj-hWpjl  +  n,  (1) 

where  is  the  response  to  the  ijk^  cell 
in  the  3-way  factorial  design  (responses 
might  be  averaged  over  those  SMEs  whose 
data  exhibit  the  same  factor  effects  (e.g., 
those  in  Figure  1),  Sy  Sv  are  the 

scale  values  associated  with  the  i^  level  of 
Coverage,  the  j^  level  of  Timeliness,  and 
the  k^^  level  of  Precision,  respectively 
(these  would  be  the  scale  values  or  mea¬ 
sures  attributed  as  SMEs  if  this  model 
accounted  for  their  data),  and  Wq  Wj 
and  Wp  are  the  weights  associated  with 
these  factors;  m  and  n  are  linear  scaling 
constants. 

If  one  did  not  already  know  that  the 
data  shown  in  Figure  1  violated  the  predic¬ 
tions  of  the  model  of  Equation  1,  the  model 
(See  VALIDITY  ISSUE,/?.  33) 


A:  Detection  B:  Classification  C:  Recognition 


Figure  1:  Hypothetical  Judgment  Data 
Observed  interactions  would  lead  to  rejecting  the  class  of  additive  models 
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Some  Fallacies  in  Cost-Risk  Analysis 


Point  estimates 
of  cost  for  new 
weapon  sys¬ 
tems  are  certain  to  be 
incorrect.  Interval 
estimates,  which  give 
a  range  of  values  with 
associated  probability, 
provide  the  decision¬ 
maker  with  much 
more  information. 
Thus  “cost-risk” 
analysis  is  supplanti¬ 
ng  conventional  cost  analysis  in  the  evalua¬ 
tion  of  new  weapon  systems.  Unfortunate¬ 
ly,  cost-risk  analysis  introduces  statistical 
considerations  that  are  unfamiliar  to  many 
cost  analysts.  As  a  consequence,  several 
serious,  persistent  errors  have  plagued  this 
emerging  field.  The  purpose  of  this  article 
is  to  expose  a  few  of  these  errors  in  the 
hopes  of  inoculating  participants  in  the 
field.  All  of  the  errors  discussed  in  this 
article  were  found  in  papers  identified 
through  a  Defense  Technical  Information 
Center  (DTIC)  search.  However,  to  mini¬ 
mize  embarrassment,  none  of  the  authors 
will  be  cited  by  name. 

Fallacy  #1:  Similarly-skewed  cost  distri¬ 
butions  must  have  non-negative  correla¬ 
tion;  oppositely-skewed  cost  distribu¬ 
tions  must  have  non-positive  correlation. 

Program  costs  are  accumulated  in  a 
hierarchical  system  of  accounts  known  as  a 
work  breakdown  structure  (WBS).  The 
cost  of  each  WBS  element  may  be  repre¬ 
sented  as  a  probability  distribution.  It  has 
been  asserted  that  two  right-skewed  cost 
elements  must  have  positive  or  zero  corre¬ 
lation;  two  left-skewed  cost  elements  must 
also  have  positive  or  zero  correlation;  but  a 
right-skewed  cost  element  and  a  left- 
skewed  cost  element  must  have  negative  or 
zero  correlation. 

It  is  easy  to  disprove  these  assertions. 
For  example,  the  exponential  distribution  is 
right-skewed.  There  are  many  families  of 
bivariate  exponential  distributions  for 
which  a  negative  correlation  is  admissible. 
Johnson  and  Kotz  (Chapter  41.3)  enumer¬ 
ate  the  following  distributions: 

•  Cumber  s  first  bivariate  exponential  dis¬ 
tribution,  correlation  as  low  as 
-.40365; 


•  GumbeTs  second  bivariate  exponential 
distribution,  correlation  as  low  as  -.25; 

•  Freund’s  bivariate  exponential  distribu¬ 
tion,  correlation  as  low  as  -.333. 

As  yet  another  example,  Johnson  and 
Kotz  (p.  20)  develop  a  bivariate  lognormal 
distribution  by  exponentiating  the  marginal 
distributions  from  a  bivariate  normal  distri¬ 
bution.  The  correlation  of  the  resulting 
lognormal  distribution  depends  on  the  vari¬ 
ances  and  correlation  of  the  underlying 
normal  variables.  When  the  normal  vari¬ 
ables  have  correlation  of  -1.0  and  respec¬ 
tive  standard  deviations  of  0.2,  the  result¬ 
ing  bivariate  lognormal  distribution  has 
correlation  -.96. 

To  make  these  examples  even  more  rel¬ 
evant  to  cost  analysis,  I  now  consider  prob¬ 
ability  distributions  specified  in  triangular 
form  (probably  the  most  common  distribu¬ 
tion  used  in  cost  analysis).  The  user  pro¬ 
vides  the  low,  high,  and  most-likely  (i.e., 
modal)  values,  and  the  analyst  linearly 
interpolates  the  density  function  between 
these  three  points.  If  the  low  and  most- 
likely  values  coincide,  then  we  have  a 
right-skewed  right  triangle;  if  the  high  and 
most-likely  values  coincide,  then  we  have 
a  left-skewed  right  triangle. 

I  will  now  provide  some  empirical 
counter-examples  to  the  fallacious  asser¬ 
tion.  Consider  two  identical,  right-skewed 
triangles  each  with  low  value  /=:10,  mode 
m=10,  and  high  value  h=50.  Using 
Microsoft  Excel,  I  first  generated  1,000 
random  draws  independently  from  each 
distribution,  so  that  the  (theoretical)  corre¬ 
lation  between  the  two  distributions  is 
equal  to  zero.  I  did  so  by  first  drawing  ran¬ 
dom  uniform  deviates  (w),  then  transform¬ 
ing  to  triangular  deviates  (jc)  using  the 

If  0<u<{m- l)f(h-l),  set  x  =l  +  ^{h- [)(m-l)u  ; 
if  (m -l)/{h- 1)  <  u  <1,  set  x=  h  —  <f{h~l){h  -m){\  -«) . 

inverse  of  the  triangular  cumulative  distrib¬ 
ution  function: 

Displayed  next  are  the  histograms  of  the 
two  triangular  distributions  that  I  generat¬ 
ed,  each  from  a  sample  of  size  1 ,000.  The 
empirical  correlation  between  the  two  dis¬ 
tributions  is  equal  to  -0.004.  The  empiri¬ 
cal  correlation  differs  slightly  from  the  the¬ 
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oretical  correlation  of  0.0,  but  this  differ¬ 
ence  would  vanish  in  a  large  enough  sam¬ 
ple. 
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To  construct  a  counter-example,  I 
returned  to  my  Excel  spreadsheet,  I  sorted 
the  1,000  values  of  Triangular  Variable  #1 
in  ascending  order,  and  I  sorted  the  1,000 
values  of  Triangular  Variable  #2  in 
descending  order.  The  two  histograms 
remain  exactly  the  same  as  pictured  above. 
But  the  sorting  serves  to  associate  high  val¬ 
ues  of  one  variable  with  low  values  of  the 
other,  inducing  a  negative  correlation.  In 
fact,  the  empirical  correlation  after  sorting 
is  equal  to  -0.939. 

As  a  second  counter-example,  I  now 
consider  a  right-skewed  triangle  with  low 
value  /=10,  mode  m=10,  and  high  value 
/z=50;  and  a  left-skewed  triangle  with  low 
value  /=10,  mode  m=50,  and  high  value 
/z=50.  Following  the  same  procedure  as 
above,  I  first  generated  1,000  random 
draws  from  the  two  distributions  by  trans¬ 
forming  independent  uniform  variates. 
The  theoretical  correlation  is  0.0,  and  the 
empirical  correlation  is  0.029.  The  two 
histograms  are  pictured  below. 


Triangular  #3 


0.15  -r 
0.10  4- 
0.05 
0.00 
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Frequenc^O  14  18  22  26  30  34  38  42  46  50 


Cost 


PHALANX 


10 


September  1996 


Triangular  #4 


0.15 

0.10 

0.05 

0.00 


Frequenc\jQ  22  26  30  34  38  42  46  50 


Cost 


To  construct  a  counter-example,  I 
returned  to  my  Excel  spreadsheet.  I  sorted 
the  1,000  values  of  Triangular  Variable  #3 
in  ascending  order,  and  I  sorted  the  1,000 
values  of  Triangular  Variable  #4  also  in 
ascending  order.  The  two  histograms 
remain  exactly  the  same  as  pictured  above. 
But  this  time,  the  sorting  serves  to  associ¬ 
ate  high  values  of  one  variable  with  high 
values  of  the  other,  inducing  a  positive  cor¬ 
relation.  In  fact,  the  empirical  correlation 
after  sorting  is  equal  to  +0.934. 


Fallacy  #2:  If  Corr(A,B)>0  and 
Corr(A,C)>0,  then  Corr(B,C)>0; 
if  Corr(A,C)<0  and  Corr(B,C)<0,  then 
Coit(A^)>0. 

The  rationale  seems  to  be  that  if  cost 
elements  B  and  C  each  move  in  the  same 
direction  as  cost  element  A,  then  they  must 
move  in  the  same  direction  as  each  other. 
Conversely,  if  cost  elements  A  and  B  each 
move  in  the  opposite  direction  from  cost 
element  C,  then  they  must  move  in  the 
same  direction  as  each  other.  These  asser¬ 
tions  are  sometimes  expressed  in  tabular 
form: 


Case 

Corr 

(A,B) 

Corr 

(A,C) 

Corr 

(B,C) 

Possible? 

I 

+ 

+ 

+ 

Yes 

n 

+ 

+ 

- 

No 

m 

+ 

- 

- 

Yes 

IV 

— 

— 

— 

No 

A  correlation  matrix  is  logically  consis¬ 
tent  if  it  is  positive  semi-definite  (see 
Marsaglia  and  Olkin);  a  necessary  and  suf¬ 
ficient  condition  is  that  the  eigenvalues  all 
be  non-negative.  Here  is  an  example  to 
illustrate  that  Case  II  is  indeed  possible: 


Variable  A 

B 

C 

A 

LO 

0.25 

0.14 

B 

0.25 

1.0 

-0.56 

C 

0.14 

-0.56 

1.0 

The  eigenvalues  of  this  matrix  are  all 
positive  (0.33,  1.10,  1.57),  indicating  posi¬ 
tive  definiteness.  Positive  definiteness 
may  also  be  tested  using  principal  minors, 
which  are  much  easier  to  compute  than 
eigenvalues.  One  merely  checks  that  the 
2x2  and  3x3  sub-determinants  (in  general, 
through  the  kxk  determinant)  anchored  in 
the  northwest  of  the  matrix  are  both  posi¬ 
tive: 


LO  025 
0.25  1.0 


=  9375  >0, 


1.0  0.25 

025  LO 
0.14  -056 


0.14 

-056|=5651>0 
1.0 


Here  is  an  example  to  illustrate  that  Case 
rV  is  also  possible: 

Variable 

A 

B 

C 

A 

1.0 

-0.12 

-0.43 

B 

-0.12 

1.0 

-0.61 

C 

-0.43 

-0.61 

1.0 

The  eigenvalues  of  this  matrix  are  all 
positive  (0,19,  1.11,  1.69),  again  indicating 
positive  definiteness.  Alternatively,  both 
principal  minors  are  also  positive: 


1.0 

-0.12 


-0.12 

1.0 


=  9856>0, 


1.0  -0.12  -0.43 

-0.12  LO  -0.61 
-0.43  -0.61  LO 


=3656  >0 


Fallacy  #3:  Maximum  negative  depen¬ 
dence  is  achieved  by  setting  all  (off-diag¬ 
onal)  correlations  equal  to  -1.0. 

When  the  correlations  are  unknown, 
sensitivity  analysis  can  be  used  to  bound 
the  results  of  a  cost-risk  analysis.  Some 
rather  wide  bounds  are  obtained  by  consid¬ 
ering,  respectively,  the  cases  of  maximum 
positive  dependence  and  maximum  nega¬ 
tive  dependence  among  the  cost  elements. 
The  former  case  obtains  when  every  (off- 
diagonal)  correlation  is  set  equal  to  +1.0. 
This  case  leads  to  the  maximum  possible 
variance  of  the  sum  of  all  the  cost  ele¬ 
ments.  It  has  been  asserted  that  the  latter 
case  (i.e.,  maximum  negative  dependence, 
implying  the  minimum  possible  variance 
of  the  sum)  obtains  when  every  (off-diago¬ 
nal)  correlation  is  set  equal  to  -1.0. 

In  fact,  when  the  off-diagonal  correla¬ 
tions  in  a  kxk  correlation  matrix  are  all 
equal,  they  are  bounded  above  by  the  con¬ 
stant: - \l{k-\)  [see  Johnson  and  Kotz, 


p. — 51].  Thus  for  a  3x3  matrix,  the  nega¬ 
tive  correlations  cannot  exceed  -0.5  in 
magnitude;  for  a  4x4  matrix,  they  cannot 
exceed  -0.33,  etc.  As  the  number  of  cost 
elements  grows  large,  a  common  negative 
correlation  is  essentially  ruled  out, 
although  a  few  isolated  negative  correla¬ 
tions  may  persist. 

When  these  conditions  are  violated,  the 
correlation  matrix  is  no  longer  positive 
semi-definite,  and  it  becomes  possible  to 
find  linear  combinations  of  cost  elements 
with  negative  “variance.”  To  illustrate  this 
point,  consider  a  3x3  matrix  with  correla¬ 
tions  equal  to  -1.0.  The  principal  minors 
reveal  a  failure  of  positive  definiteness: 


1.0 

-1.0 

1.0 

-LO 

-LO 

=  0.0, 

-1.0 

1.0 

-LO 

-1.0 

1.0 

-1.0 

-1.0 

LO 

=  -4.0 


To  further  demonstrate  the  impossibility 
of  this  situation,  suppose  that  the  three  cost 
elements  each  have  zero  mean  and  unit 
variance.  Then  the  sum  of  the  cost  ele¬ 
ments  (i.e.,  total  cost)  has  negative  vari¬ 
ance: 

Var{X,+X^+Xy)  =  {\  1  \)CorriX)  1 


M  -1  -0 

4^ 

=0  1  1) 

-1  1  -1 

-1  \j 

I 

When  the  correlations  are  equal  to  +1.0, 
all  of  the  cost  elements  move  in  the  same 
direction,  reinforcing  each  other  and  there¬ 
by  maximizing  the  variance  of  the  sum.  It 
might  be  thought  that  correlations  of  -1.0 
would  have  the  opposite  effect,  minimizing 
the  variance  of  the  sum.  To  see  why  this  is 
not  the  case,  consider  three  cost  elements 
A,  B,  and  C.  If  Corr(A,B)  =  -1.0,  then  ele¬ 
ments  A  and  B  move  in  exactly  opposite 
directions,  tending  to  offset  each  other  and 
reduce  the  variance  of  the  sum.  Similarly, 
if  Corr{B,C)  =  -1.0,  then  elements  B  and 
C  also  move  in  exactly  opposite  directions, 
again  tending  to  reduce  the  variance  of  the 
sum.  However,  this  pair  of  correlations 
imply  that  elements  A  and  C  move  in 
exactly  the  same  direction,  so  that 
Corr{A,C)  cannot  be  equal  to  -1.0;  in  fact, 
fixing  the  other  two  correlations, 
C(?rr(A,C)  must  be  equal  to +1.0.  Alterna¬ 
tively,  if  the  three  correlations  are  to  have  a 

(5ee  COST-RISK,  p.  72 
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COST-RISK 

(continued  from  p.  11) 

common  value,  this  value  cannot  exceed 
-0.5  in  magnitude. 

Fallacy  #4:  The  formula  for  aggregat¬ 
ing  variances  is  valid  only  in  the  case  of 
two-tailed  distributions. 

Although  not  restricted  to  this  context, 
we  often  think  of  the  variance  in  terms  of 
symmetric  distributions  like  the  normal,  for 
which  we  can  make  statements  like:  “X 
percent  of  the  probability  lies  within  ±Y 
standard  deviations  of  the  mean.”  In  addi¬ 
tion,  variances  may  be  aggregated  using 
the  following  formula: 

Var(h  +  B)  =  Var{A)  +  Var{B)+  2xCov{A,  B) 

=  Var{A)  +  Var{B)+  2xCorr{A, B)x^Var{A)xVar{B) . 

If  the  correlation  equals  zero,  then  (tak¬ 
ing  square  roots  of  the  equation  above)  we 
find  that  standard  deviations  may  be  aggre¬ 
gated  using  the  “root  sum-of-squares” 
(RSS)  formula: 

(7(A  +B)  =  yla\A)  +  <7\B) 

These  aggregation  formulae  and, 
indeed,  the  very  notion  of  variance  were 
questioned  in  a  paper  released  by  one 
defense  think-tank.  The  argument  is  that, 
once  contract  awards  have  been  made  and 
budgets  have  been  allocated,  costs  below 
the  budgeted  level  will  never  be  realized. 
Instead,  contractors  will  spend  at  least  their 
budgets  and  then  some.  Thus  the  ex  ante 
symmetric  cost  distributions  are  replaced 
by  ex  post  one-tailed  distributions.  More¬ 
over,  according  to  this  argument,  the 
notion  of  variance  and  the  aggregations 
formulae  are  invalidated  because  there  are 
no  longer  any  negative  deviations  (i.e., 
under-runs)  to  balance  the  positive  devia¬ 
tions  (i.e.,  over-runs).  To  replace  the  usual 
aggregation  formulae,  the  author  recom¬ 
mends  (without  any  derivation)  the  follow¬ 
ing  alternative: 

G{A+B)  =  a{A)  +  G{B) 

Under  this  alternative  formula,  the  stan¬ 
dard  deviations  rather  than  the  variances 
are  added. 

These  assertions  are  completely  false, 


and  the  alternative  formula  is  incorrect. 
The  derivation  of  the  conventional  (i.e., 
correct)  aggregation  formulae  in  no  way 
relies  upon  symmetiy  of  the  distribution;  it 
requires  only  that  the  component  variances 
exist.  The  source  of  confusion  seems  to  be 
as  follows.  The  ex  ante  symmetric  cost 
distributions  have  larger  variance  than  the 
ex  post  one-tailed  distributions.  Aggregat¬ 
ing  the  variances  of  the  ex  ante  distribu¬ 
tions  (by  the  conventional  formula)  indeed 
overstates  the  variance  of  ex  post  total  cost. 
But  the  solution  is  not  to  amend  the  con¬ 
ventional  aggregation  formula.  Rather,  the 
solution  is  to  apply  the  conventional  for¬ 
mula  to  the  variances  of  the  ex  post  distrib¬ 
utions. 

Finally,  I  will  now  show  that  the  alter¬ 
native  formula,  which  purports  to  represent 
the  zero-correlation  case,  is  in  fact  correct 
only  in  the  rather  opposite  case  in  which 
the  correlation  is  equal  to  -i-l.O.  For  in  that 
case  (and  only  that  case),  the  conventional 

(j\AiB)  =  g\A)  +  g\B)  +  2x(T(A)xff(B) 

=  [(7(A)+(T(S)f, 

(i.e.,  correct)  formula  for  Var(A-\-B) 
reduces  to: 

which  is  equivalent  to  the  alternative  for¬ 
mula. 

Fallacy  #5:  The  variance  of  an  expres¬ 
sion  can  be  reduced  by  algebraically 
grouping  the  terms. 

This  assertion,  which  I  found  in  a  mas¬ 
ter’s  thesis,  is  by  far  the  most  egregious 
abuse  of  statistics  that  I  have  ever  encoun¬ 
tered.  The  author  considers  the  cost  of 
training  three  people.  He  assigns  an  aver¬ 
age  cost  denoted  TC  (training  cost)  to  each 
person,  so  that  total  cost  is  equal  to: 

Cost  =  TC  -I-  TC  +  TC  =  3xTC. 

He  then  computes  the  variance  of  total  cost 
in  two  different  ways,  reaching  apparently 
different  results: 

Var(TC  +  TC-HTC)  = 

VariTQ  +  VariTC)  +  VariTC)  =  3iVar(TC); 
Var{3(TC)  =  9(VariTCy 

The  author  then  recommends  collecting 
costs  on  an  additive  rather  than  multiplica¬ 


tive  basis,  because  the  resulting  variance  is 
(apparently)  reduced  by  a  factor  of  three. 

This  assertion  is  ludicrous  on  its  face, 
because  one  cannot  reduce  the  variance  of 
an  expression  by  simply  grouping  the 
terms.  The  source  of  confusion  is  whether 
the  three  random  variables  labeled  “TC” 
(unfortunately,  without  subscripts)  are 
always  identical  in  value,  or  merely  identi¬ 
cally  distributed.  If  they  are  identically 
distributed  and  uncorrelated,  then  the  first 
expression  for  the  variance  is  correct.  The 
second  expression  is  not  valid  in  this  case 
because  total  cost  equals  TCj  -i-  TC2  + 
TC3,  which  cannot  be  written  as  3xTC. 

If  the  three  random  variables  are  identi¬ 
cal  in  value,  then  we  have: 

TCi  +TC2  +  TC3  =  TC  +  TC  +  TC  =  3xTC. 

Thus  the  second  expression  for  the  vari¬ 
ance  is  correct.  The  first  expression  is 
incorrect  in  this  case  because  it  ignores  the 
(perfect)  correlations  among  the  cost  ele¬ 
ments  {TC,  TC,  TC}.  Incorporating  the 
correlations  yields  the  correct  solution: 

VariTC +  TC  +  TC)  =  War{TC)  +  Var(TC)  + 
VariTC)  A-  6(Cov(TC,TC) 

=  VariTC)  +  VariTC)  +  VariTC)  + 
eiVariTC) 

=  9{VariTC). 

References 

Norman  L.  Johnson  and  Samuel  Kotz,  Dis¬ 
tributions  in  Statistics:  Continuous  Multi¬ 
variate  Distributions,  New  York:  Wiley, 
1970. 

G.  Maraglia  and  I.  Olkin,  “Generating  Cor¬ 
relation  Matrices,”  SIAM  Journal  on  Scien¬ 
tific  and  Statistical  Computing,'"  Vol.  5, 
1984,  pp.  470-475.0 


Drop  in  on  the 

NEW  AND  IMPROVED 
MORS  HOME  PAGE. 

http://www.msosa. 

miUntennet/mors/ 


PHALANX 


12 


September  1996 


Advanced  Distributed  Simulations  for  Analysis  (ADSA  ’96)  Workshop 


Introduction. 

Advanced  Distributed 
Simulations  (ADS) 
and  their  appropriate 
use  by  the  defense 
analytical  community 
were  the  focus  of  a 
recent  MORS  Work¬ 
shop,  ADSA  ’96, 
sponsored  by  ODUSA 
(OR),OCNO(N81), 
HQ  USAF/XOM, 
Joint  Staff  (J8)  and  OSD  (PA&E).  ADSA 
’96  was  another  in  a  long  series  of  MORS 
Workshops  whereby  MORS  leadership 
provides  a  forum  for  experienced  military 
analysts  from  all  services  and  the  support¬ 
ing  civilian  communities  to  weigh  in  on  a 
tough  question  or  issue  facing  the  defense 
community.  Generally,  these  workshops 
take  the  form  of  working  sessions  where 
participants  strive  to  clarify  the  question  or 
issue  and  propose  approaches  to  assist  the 
DoD  in  issue  resolution.  These  forums 
have  served  the  defense  community  well 
by  providing  insight  as  to  where  DoD  ana¬ 
lytical  efforts  should  be  focused  or  lever¬ 
aged,  or,  in  some  cases,  to  help  the  analyti¬ 
cal  community  better  understand  major 
changes  in  policy  affecting  their  work. 

For  this  particular  Workshop,  over  100 
senior  military  and  civilian  analysts  gath¬ 
ered  in  Williamsburg,  VA,  from  30  Janu¬ 
ary  to  1  February  1996  to  have  a  frank  dis¬ 
cussion  on  the  applicability/utility  of  ADS 
and  Distributed  Interactive  Simulation 
(DIS)  technologies  to  the  analytical  com¬ 
munity  and  to  document  findings  and  rec¬ 
ommendations  resulting  from  the  discus¬ 
sion.  Much  of  the  discourse  that  preceded 
the  Workshop  and  continued  through  the 
initial  workshop  sessions  focused  on  the 
need  for  a  definition  for  ADS.  Ed  Brady, 
the  General  Chair,  believed  that  any  defini¬ 
tion  would  restrict  the  potential  applicabili¬ 
ty  of  ADS  but  finally  proffered  the  follow¬ 
ing: 

Advanced  Distributed  Simulation 
—  The  evolving  DoD  distributed 
modeling  and  simulation  infra¬ 
structure,  including  synthetic  envi¬ 
ronments,  run-time  infrastruc¬ 
tures,  and  connected  human-in- 
the  loop  simulations  such  as  DIS. 

For  the  purposes  of  this  discourse,  both 


ADS  and  DIS  are  distributed  but  ADS  is 
considered  a  broader  set  of  modem  model¬ 
ing  and  simulation  capabilities  than  DIS. 
For  example,  ADS  has  the  potential  to 
include  the  interaction  of  constmctive-to- 
constmctive  simulations  and  constmctive- 
to-live  simulations  as  well  as  the  virtual 
simulations  connected  by  DIS.  Note  also 
that  this  definition  does  not  require  a 
human-in-the  loop  as  part  of  any  particular 
instance  of  an  ADS.  The  variability  of 
humans-in-the  loop  is  a  major  concern  to 
analysts  dependent  upon  replicability  of 
experiments 

Leveraging  off  lessons  learned  from 
previous  MORS  Workshops,  advance 
preparation  for  the  ADSA  ’96  included  a 
facilitators  training  session  for  chairs  and 
co-chairs  in  Arlington,  VA;  scheduled  in 
between  the  multiple  snow  storms  that 
blanketed  Washington  most  of  this  past 
winter.  This  training  session  was  to 
acquaint  the  chairs  and  co-chairs  with 
known  ADS  issues  and  to  prepare  them  for 
getting  the  most  out  of  their  respective 
working  groups.  In  addition,  all  Workshop 
participants  were  provided  a  healthy  read¬ 
ing  package  in  advance  of  the  actual  Work¬ 
shop. 

Workshop  Charge.  During  his  com¬ 
ments  at  the  opening  of  ADSA96,  Ed 
Brady  emphasized  to  the  workshop  partici¬ 
pants  that  they  should  focus  on  identifying 
potential  solutions  that  would  be  of  value 
to  both  the  analytical  community,  and  the 
modeling  and  simulation  community.  In 
particular,  the  participants  were  asked  to 
identify: 

(1)  appropriate  uses  of  ADS/DIS  capabili¬ 
ties  by  the  analytical  community, 

(2)  inherent  limitations  and  advantages  of 
ADS/DIS  capabilities  for  the  analyti¬ 
cal  community, 

(3)  current  shortcomings  of  ADS/DIS 
capabilities  that  if  remedied  could  sig¬ 
nificantly  improve  the  utility  of 
ADS/DIS  capabilities  to  the  analytical 
community,  and 

(4)  ways  to  analyze  and  test  the  relevance 
and  quality  of  ADS/DIS-based  tools. 

Products  resulting  from  the  Workshop 
were  to  be  focused  on  clarification  of 
issues  and  problems,  and  identification  of 
potential  solution  spaces.  In  particular. 


MORS  sponsors’  workshop  expectations 
included  understanding  ways  to  use  ADS- 
based  analysis  that  differed  from  those  of 
other  analyses  as  well  as  identification  of 
classes  of  analyses  most  amenable  to  sup¬ 
port  by  ADS -based  tools.  And  those  for 
which  ADS  tools  are  not  useful  would  also 
be  very  useful  to  MORS  sponsors.  This 
required  the  participants  to  think  back  on 
why  we  analyze  what  we  analyze.  Is  it 
because  of  available  analysis  tools?  Our 
own  limited  knowledge?  How  can  ADS- 
based  tools  be  used  to  give  us  more 
insight?  More  importantly,  giving  the 
changing  nature  of  the  questions  military 
analysts  are  being  required  to  answer,  what 
class  of  questions  looks  amenable  to  analy¬ 
sis  by  ADS-based  tools  independent  of 
whether  or  not  they  are  currently  available 
or  technologically  feasible?  Clearly,  an 
answer  to  this  last  question  would  help 
focus  ongoing  research.  Another  area  of 
particular  interest  is  the  identification  of 
experiments  designed  to  assist  in  the  devel¬ 
opment  of  measures  of  merit  for  analysis 
of  command  and  control,  and  other  heavily 
human-related  functions.  Given  the  evolv¬ 
ing  nature  of  ADS  and  the  changing  ques¬ 
tions  being  put  to  analysts,  there  was  room 
in  the  participants’  discussions  for  many 
differing  views  of  ADS  and  areas  ripe  for 
its  potential  use. 

The  remaining  ADSA  ’96  Workshop 
format  consisted  of  (1)  an  overview  of 
ADS  Technology  by  Dr.  Randy  Garrett 
of  DARPA,  (2)  a  keynote  address  by  Dr. 
Anita  K.  Jones  Director  of  Defense 
Research  and  Engineering,  OSD,  (3)  a 
senior  practitioners  panel,  and  (4)  the  kick¬ 
off  of  working  group  sessions.  The  work¬ 
ing  group  sessions  continued  through  the 
last  day  and  concluded  with  Working 
Group  Chairs  reporting  out  the  results  of 
their  sessions. 

Workshop  Synthesis.  ADSA  ’96  Pro¬ 
ceedings  are  being  developed  that  will  con¬ 
tain  detailed  reports  from  each  Working 
Group.  But  for  the  purposes  of  this  article, 
we  will  draw  upon  The  Synthesis  Panel 
Report  developed  by  Stuart  Starr  and  his 
panel  members.  These  panel  members 
participated  as  active  members  of  one  of 
the  other  six  working  groups  of  the  Work¬ 
shop  as  well  as  the  Synthesis  Panel. 

(See  ADSA  ’96,  p.  14) 
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ADSA  ’96 

(continued  from  p.  13) 

The  nature  of  the  problem.  Several  of 
the  Working  Groups  spent  considerable 
time  identifying  and  discussing  why  the 
analysis  community  has  been  slow  to 
embrace  ADS.  These  issues  can  be  loosely 
aggregated  into  two  categories:  quality  and 
use. 

In  the  area  of  quality,  the  Working 
Groups  cited  the  following  concerns 
among  others: 

•  Ambiguity.  Since  many  of  the  members 
of  the  analysis  community  are  not  sure 
what  ADS  is,  they  are  not  sure  how  use¬ 
ful  it  is  to  them. 

•  Replicability.  Since  many  examples  of 
ADS  feature  humans-in-the-loop,  there  is 
concern  among  some  analysts  about  the 
ability  to  replicate  results.  In  particular, 
there  is  concern  about  coping  with  the 
effects  of  subject  learning  and  the  impact 
of  differences  in  subject  background, 
experience,  proficiency,  and  morale. 

•  Credibility.  It  was  acknowledged  by  sev¬ 
eral  Working  Groups  that  verification, 
validation,  and  accreditation  (VV&A)  of 
ADS  pose  particular  challenges. 

In  the  area  of  use,  the  Working  Groups 
cited  the  following  obstacles  among  others: 

•  Cost/resources  implications.  Several 
Working  Groups  observed  that  it  is  like¬ 
ly  to  be  relatively  expensive  to  acquire, 
update,  and  employ  ADS  in  comparison 
to  more  traditional  analysis  tools.  In 
addition,  from  a  resource  perspective,  it 
is  often  difficult  to  obtain  and  train  suit¬ 
able  test  subjects. 

•  Time/schedule  implications.  Numerous 
Working  Groups  expressed  concern 
about  the  extensive  time  implications  of 
ADS.  This  includes  the  time  to  create  an 
ADS  capability,  set-up  conditions  for  a 
specific  analysis,  perform  test  runs,  and 
reduce  the  data  that  is  generated  by  exer¬ 
cising  an  ADS.  These  time  considera¬ 
tions  can  preclude  use  of  ADS  when  the 
analyst  is  given  little  lead  or  execution 
time  (e.g.,  Program  Objective  Memoran¬ 
dum  issue  analyses). 

•  Data  implications.  Several  participants 
identified  a  broad  spectrum  of  data  issues 
associated  with  ADS.  These  include 
problems  associated  with  acquiring  data 
needed  to  set-up  and  execute  ADS; 
deciding  what  data  to  collect  and  collect¬ 


ing  it  (note:  it  may  be  difficult  to  intro¬ 
duce  ad  hoc  “hooks”  into  an  ADS  to  col¬ 
lect  desired  data);  and  storing  and  pro¬ 
cessing  the  resulting  data  (e.g.,  some  of 
the  data  of  interest  may  be  spoken  by  the 
test  subjects). 

•  Efficient,  effective  exercising  of  ADS.  In 
order  to  employ  ADS  efficiently  and 
effectively,  it  is  necessary  to  devise  and 
implement  experimental  designs  that 
provide  confidence  levels  consistent  with 
the  issues  under  study  and  cope  with 
potential  confounding  factors  that  may 
be  unique  to  ADS  (e.g.,  compensate  for 
potential  learning  by  test  subjects). 

In  view  of  the  many  issues  associated 
with  ADS,  the  most  natural  question  is: 
“Why  should  analysts  bother  with  ADS?” 
The  Workshop  identified  several  signifi¬ 
cant  opportunities  implicit  within  ADS  that 
suggest  that  it  is  well  worth  the  time  of  the 
analysis  community  to  explore  the  applica¬ 
tion  of  ADS  to  a  broad  range  of  analyses. 

•  ADS  has  the  potential  to  enhance  sub¬ 
stantially  analytic  support  to  a  number  of 
application  areas. 

•  ADS  may  enable  analysts  to  address 
issues  in  a  richer  context  than  they  other¬ 
wise  could. 

•  Information  technology  trends  are  provid¬ 
ing  new  opportunities  and  are  being 
enhanced  dramatically  by  the  commer¬ 
cial  sector. 

•  The  analytic  community  has  the  opportu¬ 
nity  to  take  advantage  of  the  extensive 
ADS  investments  of  other  communities 
(most  notably  the  training  community). 

•  Several  senior  practitioners  postulated 
that  ADS  poses  the  opportunity  for  ana¬ 
lysts  to  move  to  “computer  assisted 
analysis”  and  thereby  “return  to  our 
roots.”  They  also  observed  that  decision 
makers  are  being  exposed  to  ADS 
through  its  application  by  other  commu¬ 
nities  and  will  come  to  expect  its  attrac¬ 
tive  characteristics. 

•  Jack  Thorpe,  SAIC,  predicted  that  “In 
ten  years,  everything  can  be  connected  to 
everything  else  and,  given  the  spirit  of 
experimentation  that  exists  within  the 
analytic  community,  it  will! 

Findings  and  recommendations. 

Based  on  the  insights  that  the  Synthesis 
Panel  gained  from  participation  on  Work¬ 
ing  Groups  and  Synthesis  Panel  discus¬ 
sions,  several  areas  were  identified  that  the 


panel  members  felt  were  of  overarching 
importance.  For  each  of  these  areas,  the 
Synthesis  Panel  developed  findings  and 
recommendations.  As  these  are  initial 
findings  and  recommendations,  they  are 
subject  to  modification,  clarification,  and 
addition  as  the  Proceedings  are  completed 
based  upon  final  Working  Group  Reports. 

Community  Leadership.  The  Synthe¬ 
sis  Panel  found  that  the  challenges  associ¬ 
ated  with  ADS  transcend  the  abilities  and 
resources  of  the  individual  analyst.  If  ana¬ 
lysts  are  to  use  ADS  effectively,  a  para¬ 
digm  shift  must  occur  to  enhanced  collabo¬ 
rative  analysis.  This  concept,  which  was 
put  forth  by  the  Analysis  and  Require¬ 
ments  Working  Group,  rests  on  pillars  of 
cross-community  shared  data,  tools  (e.g., 
scenario  generation  tools,  visualization 
tools),  security.  Measures  of  Merit 
(MoMs),  and  an  integrated  family  of  analy¬ 
sis  techniques.  The  last  of  these  pillars 
should  be  emphasized.  Most  of  the  Work¬ 
ing  Groups  observed  that  ADS  should  not 
be  viewed  as  a  “stand-alone”  capability 
that  can  be  applied  to  all  analytic  problems. 
In  general,  it  must  be  harmonized  and 
orchestrated  with  other  analysis  techniques 
to  compensate  for  selected  ADS  features 
(e.g.,  extensive  time  to  create  and  execute) 
and  employed  where  appropriate.  In  order 
to  bring  about  this  paradigm  shift,  key  ana¬ 
lytic  organizations  and  the  analytic  com¬ 
munity  must  provide  needed  stimulation 
and  leadership.  A  partial  listing  of  recom¬ 
mended  actions  to  be  undertaken  by  the 
Services,  the  Joint  Staff,  and  OSD  include 
the  following: 

•  Provide  incentives  to  use  ADS. 

•  Make  necessary  investments  in  commu¬ 
nity  infrastructure  and  accreditation 
efforts. 

•  Facilite  needed  education,  training. 

•  Form  ADS  teams  with  the  necessary  mix 
of  skills.  Die  effective  creation  and  use 
of  ADS  demands  a  team  effort. 

ADS  Plan  for  the  Analysis  Communi¬ 
ty.  Once  the  analysis  community  becomes 
committed  to  the  enhanced  collaborative 
analysis  paradigm,  it  will  need  a  plan  to 
guide  its  actions.  Historically,  the  analysis 
community  has  been  reticent  to  take  this 
step.  For  example,  several  years  ago, 
DMSO  sponsored  a  series  of  workshops  to 
stimulate  the  various  communities  to  iden- 
(See  ADSA  ’96,  p.  21) 
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In  the  June  1996  issue  of  the  PHA¬ 
LANX,  Anita  Zabek  and  Gordon 
Miller  discussed  the  Aggregate  Level 
Simulation  Protocol  (ALSP),  a  protocol 
that  permits  multiple  simulations  to  inter¬ 
operate,  and  the  Joint  Training  Confedera¬ 
tion  (JTC).  ^  This  discussion  included  tech¬ 
nical  interoperability,  using  the  ALSP 
Infrastructure  Software  (AIS),  and  opera¬ 
tional  or  functional  interoperability. 

This  article  continues  that  discussion  by 
focusing  on  intelligence  simulation  support 
to  major  JTC  exercises.  This  includes  a 
review  of  the  intelligence  training  audi¬ 
ence;  the  intelligence  simulations  typically 
used  in  major  JTC  exercises;  and  several 
technical  architecture  issues  including 
security,  software  interfaces  and  maintain¬ 
ability;  as  well  as  managing  future  devel¬ 
opment. 

The  Joint  Training  Confederation 

A  very  brief  review  of  the  JTC  is  appro¬ 
priate  to  establish  background  for  the  dis¬ 
cussion  of  intelligence  training.  The 
Joint  Training  Confederation  is  an  ALSP- 
supported  confederation  of  simulations.  Its 
purpose  is  to  support  Service  and  joint 
training  of  senior  commanders  and  staffs. 

Simulations  that  interoperate  with  each 
other  through  the  AIS  are  members  of  the 
JTC.  Eight  simulations  currently  directly 
participate  in  the  JTC  as  shown  in  Figure  1 
Each  of  the  simulations  is  supported  by 
a  Service  or  joint  activity  and  provides  a 
representation  of  the  domain  of  particular 
interest  to  that  activity.  For  example, 
RESA  provides  the  naval  surface,  subsur¬ 
face,  and  air  training  environment. 
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Figure  1. 

The  Joint  Training  Confederation 


The  intelligence  simulation  in  the  JTC 
is  the  Tactical  Simulation  (TACSIM).  This 
interactive,  computer-based  training  simu¬ 
lation  was  specifically  designed  and  used 
to  support  intelligence  training  in  the 
Army.  In  its  role  in  the  JTC,  TACSIM  also 
supports  intelligence  training  in  joint  exer¬ 
cises. 

The  Intelligence  Training 
Audiences 

The  combat  simulations  in  the  JTC  pri¬ 
marily  support  training  of  senior  comman¬ 


ders  and  their  operations  staffs  (i.e.,  G3, 
J3).  The  combat  simulations  provide  very 
little  training  for  the  general  or  joint  intelli¬ 
gence  staffs  (G2  or  J2)  and  no  training  at 
all  for  other  intelligence  activities.  There¬ 
fore,  high  resolution,  high  fidelity  intelli¬ 
gence  simulations  have  evolved  to  meet 
the  specific  needs  of  the  intelligence  train¬ 
ing  audiences. 

Figure  2  shows  the  intelligence  training 
audiences  and  their  relationship  with  the 
intelligence  cycle.  Commanders  and  staffs 
of  intelligence  units  ensure  that  intelligence 
assets  are  available  to  support  the  mission. 
Collection  managers  allocate  requirements 
to  specific  collection  assets.  Intelligence 
collectors  perform  the  collection  process. 
Intelligence  analysts  process  raw  intelli¬ 
gence  and  produce  intelligence  products 
that  are  disseminated  to  commanders  and 
staffs.  Commanders  and  staffs  use  the 
intelligence  products  to  support  their  plan¬ 
ning  and  execution  process  and  generate 
requirements  for  further  intelligence. 

In  organizational  terms,  each  command 
headquarters  has  an  intelligence  staff  offi¬ 
cer,  the  S2,  G2,  or  J2,  and  supporting  staff. 

(See  INTELLIGENCE,/?  16} 


The  Training  Audience 


Intelligence  Preparation  of  the 
Battlefield  (IPD) 

•Battlefield  Area  Evaluation 
•Terrain  Analysis 
•Weather  Analysis 
•Threat  Evaluation 
•Threat  Integration 
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INTELLIGENCE 

(continued  from  p,  15) 

The  intelligence  staff  officer  advises  the 
commander  and  other  staff  sections  and 
coordinates  intelligence  support  for  the 
command.  In  addition,  most  major  com¬ 
mands  have  a  supporting  military  intelli¬ 
gence  unit  such  as  a  military  intelligence 
platoon,  company,  battalion,  or  brigade. 
These  intelligence  units,  with  their  special¬ 
ized  personnel  and  equipment,  perform  the 
collection  management,  intelligence  collec¬ 
tion,  and  intelligence  analysis  functions. 

The  role  of  the  intelligence  training 
simulations  is  to  provide  an  environment 
for  training  the  intelligence  staffs  at  the 
major  headquarters  and  the  specialized 
intelligence  functions  performed  by  per¬ 
sonnel  in  the  military  intelligence  units. 

Joint  Training  Confederation 
Intelligence  Architecture 

TACSIM  is  the  “official"  JTC  intelli¬ 
gence  simulation.  However,  to  meet  the 
demands  of  the  intelligence  training  audi¬ 
ences,  a  more  complicated  intelligence 
architecture  has  emerged  over  the  past  few 
years.  Figure  3  shows  the  major  simula¬ 
tions,  processes  and  data  flows. 

The  single  block  on  the  JTC  architec¬ 
ture  diagram  labeled  TACSIM  is  really  a 


large  collection  of  simulations  and  process¬ 
es  which  interact  to  provide  a  full  intelli¬ 
gence  portrayal.^  This  collaboration 
between  simulations  in  the  JTC  and 
numerous  specialized  “peripheral  software 
systems"  is  typical  for  simulations  in  the 
JTC. 

The  data  flow  starts  with  the  game 
truth  generated  by  the  combat  simulations 
shown  on  the  left  of  the  diagram.  Data 
flows  through  ALSP  and  the  CBS  Master 
Interface  to  the  intelligence  simulations. 

The  primary  data  flow  is  into  the  Spe¬ 
cial  Compartmented  Information  Facility 
(SCIF)  to  TACSIM,  the  National  Wargam¬ 
ing  System  (NWARS),  and  the  Synthetic 
Imagery  Generation  System  (SIGS).  These 
provide  output  to  the  training  audience  and 
to  the  TACSIM  Analysis  Operations  Node 
(TALON)  and  the  TACSIM  After  Action 
Report  Utility  System  (TAARUS). 

The  second  major  data  flow  is  through 
the  High  Resolution  Simulation  System 
(HRSS)  to  the  JSTARS  Simulation  System 
(JSS)  and  the  Multiple  UAV  Simulation 
Environment  (MUSE). 

TACSIM  is  an  Army  interactive  simu¬ 
lation  that  provides  realistic  intelligence 
products  for  the  evaluation  of  Command, 
Control,  Communications,  Intelligence, 
and  Electronic  Warfare  (C3IEW)  func¬ 
tions,  systems  development,  and  training.^ 


Figure  3. 

Typical  Exercise  Intelligence  Architecture 


TACSIM: 

•  Simulates  the  composition,  disposition, 
and  actions  of  an  opposing  force. 

•  Models  friendly,  or  blue  force,  intelli¬ 
gence  collectors  to  simulate  collection 
tasking  requested  by  participants,  play¬ 
ers,  and  testers. 

•  Produces  intelligence  reports  resulting 
from  simulated  collection  missions. 

TALON  is  an  intelligence  fusion 
process  that  emulates  the  function  of  intel¬ 
ligence  analyst  cells  when  these  cells  are 
not  participating  in  an  exercise.  Based  on 
intelligence  information  produced  by 
TACSIM,  TALON  can  provide  summary 
intelligence  reports  directly  to  player  intel¬ 
ligence  staffs  at  major  headquarters. 

TAARUS  provides  detailed,  after 
action  reports.  It  maintains  a  detailed  histo¬ 
ry  of  the  intelligence  operations  during  an 
exercise  and  can  provide  this  history  in  a 
variety  of  formats  suited  to  intelligence 
player  critiques. 

NWARS  portrays  intelligence  collec¬ 
tion  by  national  collection  assets.  This 
complements  the  simulation  of  intelligence 
collection  by  theater  assets  in  TACSIM. 

SIGS  provides  hardcopy  imagery  to 
players  based  upon  a  library  of  existing 
imagery  and  data  from  NWARS  on  exer¬ 
cise  game  truth.  Although  labor  intensive, 
SIGS  adds  significant  realism  to  exercises 
through  the  interjection  of  realistic  intelli¬ 
gence  imagery  products. 

HRSS  disaggregates  ground  units 
down  to  the  physical  entity  level  based 
upon  aggregated  unit  input  from  the 
ground  combat  simulations  and  an  internal 
rule  set. 

JSS  receives  the  entity  level  disposition 
from  HRSS  and  provides  a  J-STARS 
Moving  Target  Indicator  display. 

MUSE  is  a  UAV  operator  training  sim¬ 
ulator  in  which  a  UAV  operator  flies  a 
UAV  over  the  battlefield  and  sees  a  realis¬ 
tic  real  time  video  at  the  operator’s  control 
station.  Commanders  and  staffs  are  begin¬ 
ning  to  expect  these  real-time  videos  as  a 
normal  intelligence  product. 

Architecture  Challenges 

The  technical  interaction  of  the  intelli¬ 
gence  simulations  and  some  unique  prob¬ 
lems  inherent  in  them  pose  numerous  chal¬ 
lenges  for  developers  and  users.  Most  of 
the  challenges  result  from  the  use  of  simu¬ 
lations  to  support  training  needs  beyond 
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that  for  which  they  were  originally 
designed. 

Strict  security  requirements  add 

complexity  to  the  system.  TACSIM  and 
several  of  its  peripheral  software  systems 
operate  in  a  SCIF.  The  JTC  and  intelli¬ 
gence  simulations  outside  the  SCIF  could 
operate  at  the  unclassified  level,  though 
they  normally  operate  at  the  secret  collater¬ 
al  level. 

Three  processes  direct  the  data  flow 
into  and  out  of  the  SCIF.  The  TACSIM 
Message  Automated  Processor  (TMAP)  is 
the  process  that  collects  simulation  data 
and  directs  the  data  flow  into  the  SCIF. 
The  Communication  Support  Processor 
(CSP)  serves  as  the  interface  for  data  trans¬ 
mission  between  the  collateral  and  com- 
partmented  processing  environments.  The 
TACSIM  Relay  for  Input/Output  Processor 
(TRIOP)  is  a  message  distribution  facility 
that  routes  messages  to  and  from  the  CSP. 

The  need  to  maintain  these  processes 
requires  continuing  effort  and  imposes  bur- 
dens  on  intelligence  simulations  as 
described  in  the  following  paragraphs. 

Data  transfer  by  file  processes  are 
used  to  implement  the  above  security 
processes.  For  example,  TAT  outputs  a 
data  file  into  a  common  directory.  The  next 
process  in  the  sequence,  TMAP,  copies 
this  file  into  its  input  file  directory,  reads 
the  file,  modifies  the  file,  and  then  outputs 
a  file  to  the  next  process.  This  is  a  simple 
process,  but  also  a  potentially  slow  and  I/O 
bound  process. 

Simulation  data  flow  is  one-way  from 
the  JTC  into  the  SCIF.  There  is  no  elec¬ 
tronic  information  flow  from  within  the 
SCIF  to  the  simulations  in  the  JTC  because 
of  security  requirements.  The  products 
from  TACSIM  and  the  other  intelligence 
simulations  operating  in  the  SCIF  are  sent 
to  the  training  audience  on  real-world  intel¬ 
ligence  systems. 

This  causes  an  automatic  requirement 
for  players  or  controllers  to  integrate  the 
intelligence  with  the  decision  processes  in 
the  simulations.  There  is  no  electronic  con¬ 
nection  for  intelligence  data  flow  back  into 
the  simulations. 

Limited  simulation  time  manage¬ 
ment  is  another  result  of  the  one-way  data 
flow  from  the  JTC  to  TACSIM.  TACSIM 
can  receive  the  current  simulation  time 
from  the  JTC,  so  TACSIM  time  will  not 
get  ahead  of  JTC  time.  However,  TACSIM 
is  unable  to  send  its  current  time  back  the 
JTC,  so  there  is  the  potential  for  the  JTC  to 


get  ahead  of  TACSIM.  In  practice,  TAC¬ 
SIM  typically  runs  a  few  minutes  behind 
the  JTC,  but  this  is  usually  not  noticeable 
to  the  training  audience. 

Numerous  interface  programs  sup¬ 
port  the  data  flow.  The  number  of  these 
programs  reflects  the  evolutionary  nature 
of  the  JTC  and  the  movement  toward 
greater  interoperability  between  simula¬ 
tions. 

The  TACSIM  Interface  Program  (TIP) 
is  a  CBS  Master  Interface  program  and 
was  the  first  interface  between  CBS  and 
TACSIM. 

As  TACSIM  became  more  “joint,”  the 
TACSIM  ALSP  Translator  (TAT)  was 
developed  to  enable  coordination  of  air 
reconnaissance  flights  with  the  air  simula¬ 
tions,  and  collection  on  non- Army  ground 
units  and  ships.  The  DMMAIN  TAT  Inter¬ 
face  (DTI)  coordinates  air  reconnaissance 
flight  information  from  the  TAT  with  air 
reconnaissance  and  sensor  information  in 
DMMAIN,  a  subordinate  process  of  TAC¬ 
SIM. 

Each  of  these  processes  serves  as  a  con¬ 
duit  from  different  simulations  and  the  data 
they  convey  is  not  mutually  exclusive.  For 
example,  Blue  and  Red  unit  data  from 
CBS  may  come  through  the  TIP  while 
Blue  unit  data  from  all  simulations  may 
come  through  the  TAT.  Ground  feature 
information  on  minefields  and  similar 
man-made  features  may  only  come 
through  the  TIP  from  CBS  at  this  time. 

Multiple  tactical  intelligence  simula¬ 
tions  also  contribute  to  the  complexity. 
Unclassified  simulations,  such  as  the  Battle 
Command  Training  Program  (BCTP) 
Intelligence  Collection  Model  (BICM), 
often  perform  two  exercise  functions. 

First,  Red  players  should  have  a  realis¬ 
tic  and  appropriate  view  of  the  battlefield 
in  order  for  operational  security  planning 
and  execution  by  the  blue  players  to  con¬ 
tribute  to  the  success  or  failure  of  blue 
operations. 

Second,  an  unclassified  intelligence 
program  is  often  convenient  in  conducting 
training.  Therefore  unclassified,  low  reso¬ 
lution,  intelligence  programs,  such  as 
BICM,  are  often  used. 

However,  the  use  of  two  intelligence 
simulations  performing  essentially  the 
same  task  in  major  exercises  may  be  an 
unneeded  complexity. 

Different  operating  systems  and 
computer  platforms  contribute  to  the  soft¬ 
ware  maintenance  challenge.  The  develop¬ 


ment  cycles  for  the  major  simulations  and 
the  release  of  new  computers  and  operating 
systems  by  commercial  manufacturers 
contribute  to  the  continuing  requirements 
to  upgrade  the  interfaces  between  systems. 
As  the  number  of  intelligence  simulations 
grows,  the  difficulty  of  maintaining  the 
interfaces  increases  exponentially. 

High  TACSIM  controller  require¬ 
ments  result  from  the  initial  design  of 
TACSIM  and  the  expanded  TACSIM  role 
in  supporting  training  at  higher  command 
and  staff  levels.  For  example,  since  TAC¬ 
SIM  was  designed  to  support  training  of 
intelligence  collection  managers,  there  was 
no  provision  made  for  automated  collec¬ 
tion  management  for  exercises  in  which 
collection  managers  were  not  playing. 
Some  higher  level  command  and  staff 
exercises  do  not  have  collection  managers 
as  part  of  the  training  audience,  so  con¬ 
trollers  have  to  be  provided  to  perform  this 
function. 

Data  base  preparation  and  mirror¬ 
ing  remains  a  significant  problem.  TAC¬ 
SIM,  like  many  other  simulations,  has 
unique  data  requirements  because  of  its 
specialized  simulation  environment.  For 
example,  TACSIM  requires  intelligence 
signature  data  for  equipment  and  units  in 
the  scenario  being  played.  This  data  needs 
to  be  developed  prior  to  the  beginning  of 
an  exercise  and  this  leads  to  duplication  of 
data  between  simulations.  This  data  base 
mirroring  requires  close  coordination 
between  the  exercise  data  base  developers 
for  each  simulation. 

Managing  for  the  Future 

The  TACSIM  Project  Office  approach¬ 
es  its  mission  of  supporting  intelligence 
training  with  simulations  in  three  ways. 

First,  TPO  will  maintain  the  current 
intelligence  training  system.  Interfaces  will 
be  kept  current  and  TPO  will  remain 
responsive  to  changes  required  by  TAC¬ 
SIM  users.  This  is  essential  since  TACSIM 
will  continue  to  be  the  Army  intelligence 
simulation  of  choice  until  follow-on  sys¬ 
tems  are  developed  and  deployed. 

Second.  TPO  will  address  some  of  the 
architecture  challenges  with  TACSIM  and 
other  current  generation  intelligence  simu¬ 
lations.  As  resources  become  more  scarce, 
TPO  needs  to  make  some  modest  invest¬ 
ment  in  improving  the  maintainability  of 
the  current  system. 

(See  INTELLIGENCE,/?.  22) 
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64th  MORS  SYMPOSIUM 


64th  MORSS  Wrap  Up 


Dick  Helmuth,  SAIC,  Program  Chair 

The  64th  MORSS  at  Fort  Leaven¬ 
worth,  Kansas  is  now  in  the  memo¬ 
ry  banks,  but  what  wonderful 
memories  those  are.  The  facilities  and 
local  support,  technical  meetings,  and 
social  activities  all  merged  flawlessly  to 
provide  that  ideal  setting  for  personal  and 
professional  growth  that  we  have  come  to 
expect  from  the  annual  MORS  Sympo¬ 
sium.  A  very  large  number  of  people  com¬ 
bined  their  efforts  to  make  the  whole  event 
much  greater  than  the  sum  of  its  compo¬ 
nents  and  I  want  to  take  this  final  opportu¬ 
nity  to  acknowledge  their  contribution. 

The  cooperation  and  willingness  of 
Fort  Leavenworth  to  host  and  support  this 
event  cannot  be  overstated.  The  local 
team  led  by  Phil  Kubler  with  Sheila 
Nickings,  Dottie  Burns,  Hugo  Mayer, 
Carol  Mullen  and  Delite  Tyler  have  set 
new  standards  for  what  can  be  accom¬ 
plished.  The  goal  was  that  every  support 
function  flow  so  smoothly  that  you  don’t 
even  notice  it,  and  that  level  of  excellence 
was  achieved.  The  MORS  Office  staff, 
which  continues  to  get  better  every  year, 
contributed  their  full  share  too.  Dick 
Wiles,  Natalie  Addison,  Cynthia  Kee- 
LaFreniere,  and  Michael  Cronin  (a  great 
program!)  deserve  all  the  accolades  we 
can  give  them,  and  many  more.  Several 
other  “regulars”  contributed  to  the  smooth 
functioning  of  the  “office  away  from 
home”  to  include:  Peggy  Lambert,  Bob 
McIntyre,  Gene  Schroeder,  Anne  Pate- 
naude,  Faye  Brady,  Lounell  Southard 
and  a  “host”  of  helpers  from  the  MORS 
Board  of  Directors.  A  special  thanks  to 
Mike  Bauman  and  the  many  others  who 
helped  to  organize  the  outstanding  West¬ 
ern  Barbecue  at  the  National  Agricultural 
Center  and  Hall  of  Fame.  Everyone 
agreed  that  it  was  a  true  highlight  of  the 
week.  Also  deserving  special  thanks  are 
Jean  Bauman  and  Marilyn  Kubler  who 
organized  and  hosted  a  great  Spouse/Guest 
tour.  The  final  event,  a  trip  by  more  than 
20  to  watch  the  MLS  game  between  the 
KC  Whiz  and  the  DC  United  organized  by 
Anne  Patenaude,  was  a  fitting  and  suc¬ 


cessful  end  to  the  week. 

But,  no  matter  the  quality  of  the  sup¬ 
port,  it  is  the  technical  meetings  that  are  at 
the  heart  of  our  symposium.  General  (ret) 
David  Maddox  gave  an  outstanding 
Keynote  address  that  framed  the  sympo¬ 
sium’s  theme  of  “Leveraging  Technology 
for  the  Military  Analyst”  perfectly.  A  spe¬ 
cial  thanks  is  due  to  Stu  Starr  and  his  four 
session  chairs.  Dr.  Russ  Richards,  Iris 
Kameny,  Nahum  Gershon,  and 
Marchelle  Stahl,  for  a  superb  job  of  devel¬ 
oping  and  presenting  details  of  that  theme 
in  the  Special  Sessions.  Also  a  special 
thanks  to  Gene  Visco  and  Bob  Orlov  for 
organizing  an  excellent  Junior/Senior  Ana¬ 
lyst  Session,  and  to  Sue  Iwanski  for  orga¬ 
nizing  a  terrific  Poster  Session  at  the 
Mixer.  Also  of  special  note  is  the  great  set 
of  tutorials  organized  by  Dr.  Roy  Rice,  and 
the  other  Special  Sessions  which  made  a 
genuine  contribution  to  excellence. 

Everyone  should  recognize  that  the  real 
heart  of  every  MORS  Symposium  is  the 
work  done  in  the  Working  and  Composite 
Group  Sessions.  Kerry  Kelley,  as  Work¬ 
ing  Group/Composite  Group  Coordinator, 
had  the  year-long  task  to  make  that  happen 
well,  and  with  the  help  of  seven  great 
Composite  Group  Chairs,  Pat  McKenna, 
Bill  Mulholland,  Jay  Wilmeth,  Audree 
Newman,  John  Green,  Mary  JoAnn 
Carroll,  and  LTC  Mike  McGinnis,  32 
Working  Group  Chairs,  and  many  co¬ 
chairs,  they  made  this  segment  of  the  pro¬ 
gram  the  success  it  needs  to  be. 

It  took  a  very  large  number  of  volun¬ 
teers  to  make  the  64th  MORSS  such  a  suc¬ 
cess,  and  only  a  few  of  them  can  be  named 
in  an  article  such  as  this,  but  many  more 
deserve  our  thanks.  I  know  that  each  rec¬ 
ognizes  the  value  of  their  contribution  and 
how  it  helped  to  make  the  symposium  a 
success  for  someone  else.  That  is  our  pri¬ 
mary  reward.  Each  of  you  have  a  chance 
to  get  involved  now  with  the  65th 
MORSS.  The  program  chair  notebook  has 
been  passed  to  Dr.  Harry  Thie,  and  the 
Marine  Corps  at  Quantico  is  beginning 
preparations  already.  Don’t  miss  it!  O 


65th  MORSS  Set 
for  Quantico  in  ’97 

Harry  Thie,  Program  Chair 

Mark  your  calendar  now — June 
10,  11,  12  1997— for  the  65th 
MORSS  at  Quantico,  VA.  The 
theme  for  the  65th  is  “Analysis  for  Com¬ 
plex,  Uncertain  Times.”  This  theme  is 
appropriate  for  MORS,  for  the  Marine 
Corps,  and  for  1997. 

We  will  be  hosted  by  Marine  Corps 
University.  If  you  have  not  visited  Quanti¬ 
co  recently,  you  will  be  amazed  at  all  the 
Marine  Corps  has  underway,  and  we  will 
take  advantage  of  many  of  their  new  facili¬ 
ties  while  we  are  there.  Quantico  is  just 
south  of  Washington,  DC,  and  we  will 
accommodate  daily  atttendees  via  bus  ser¬ 
vice  to  and  from  the  Pentagon. 

You  have  stated  in  your  responses  to 
our  surveys  that  you  want  the  opportunity 
at  the  annual  symposium  to  share  expertise 
among  your  peers;  to  meet  old  and  new 
colleagues;  and  to  learn.  We  are  planning 
to  achieve  all  of  these. 

The  format  of  the  symposium  will  be 
slightly  different  from  last  year.  The 
Working  Group  Chairs  asked  that  more 
time  be  allocated  to  working  group  ses¬ 
sions  so  we  will  do  that.  We  will  have  8 
working  group/composite  group  sessions 
during  the  three  days.  Thus  you  will  have 
additional  time  to  engage  in  peer  dialogue. 
The  tutorials  on  Tuesday,  Wednesday,  and 
Thursday  will  focus  on  educational  oppor¬ 
tunities — a  chance  to  learn  some  new 
tricks  of  the  trade.  And  we  will  schedule 
social  activities  for  both  Tuesday  and 
Wednesday  evenings. 

There  is  an  energetic  group  already  at 
work  planning  and  scheduling.  Their 
names  appear  on  the  next  page.  If  you 
have  ideas,  please  contact  one  of  the  pro¬ 
gram  staff.  Their  phone  numbers  and  e- 
mail  addresses  are  shown. 

During  the  year  leading  up  to  Quantico, 
you  will  observe  some  administrative 
changes  in  the  method  and  content  of  the 
various  registration  materials.  The 
announcement  and  call  for  papers  will  be 
sent  in  October.  We  will  also  experiment 
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65th  MORSS  Program  Staff 

Responsibilities 

Name(s) 

Phone 

e-mail 

Program  Chair 

Harry  Thie 

202-296-5000x5379 

thie@rand.org 

Deputy  Chair 
(Logistics) 

A1  Brandstein 

703-784-3235 

brandsteina@  mqg-smtp3  .usmc.mil 

Deputy  Chair 
(Operations) 

Kerry  Kelley 

402-294-8117 

kelleyk@j5.stratcom.af.mil 

Deputy  Chair 
(Plans) 

RADM  Pierce  Johnson 

202-685-5014 

radm.johnson@smtpgw.msc.navy.mil 

Site  Coordinator 

COL  Richard  J.  Linhart  Jr. 

703-784-3235 

linhartr@mqg-smtp3.usmc.mil 

Plenary/Special 

Sessions  Coordinator 

Working  Group 

Composite  Group 
Coordinators 

Ted  Smyth 

Sue  Iwanski 

CAPT  Lee  Dick 

Anne  Patenaude 

Jay  Wilmeth 

Bill  Skeith 

Bob  Sheldon 

LTC  Mike  McGinnis 

Mary  Bonnet 

301-953-6342 

516-346-9138 

703-602-2791 

703-749-5109 

703-695-1763 

719-635-2571  x46 
703-558-2759 

914-938-2700 

703-695-5552 

ted.smyth@jhuapl.edu 
iwanski  @grLimman.com 
dickl@smtp-gw.spawar.navy.mil 
anne_patenaude@cpqm.saic.com 
j  wilmeth  @  is  l.js.ml 
bskeith@logicon.com 
sheldon@brick.saic.com 
fm0768  @exmail. usma.edu 
bonnet@afsaa.hq.af.mil 

Poster  Session 

Coordinators 

Ken  Martel  1 

MAJ  Willie  McFadden 
Gene  Visco 

703-845-1000 

703-697-1175 

kmartell@calibresys.com 
mcfadde  w  @  tfs.  net 
viscoep@hqda.army.mil 

Tutorials 

Coordinators 

Pat  Vye 

Dean  Hartley 

Phil  Kubler 

703-697-8590 

423-574-7670 

913-684-9164 

pvye@isl.js.mil 

dhx@oml.gov 

kublerp@trac.army.mil 

Best  Paper  Session 
Coordinator 

Jim  Duff 

804-498-5646 

duff@tecnet  1  .jcte.jcs.mil 

Education  Session 
Coordinator 

Yupo  Chan 

513-255-3362 

ychan@afit.af.mil 

VIP  Coordinator 

MAJ  Mark  Gallagher 

402-294-1656 

gal  laghm  @  j  5 .  stratcom .  af  m  i  1 

Junior/Senior  Analyst 
Coordinators 

Robert  Orlov 

Howard  Whitley 

CDR  Denny  Baer 

703-693-3248 

301-295-1611 

703-604-0307 

rorlov@isl.js.mil 

whitley@caa.army.mil 

baer-dennis@ncca.navy.mil 

Spouse/Guest 

Coordinator 

Ginny  Wiles 

703-751-7290 

ginnywiles@aol.com 

MORS  Staff 

Dick  Wiles 

Natalie  Addison 

Cynthia  Kee-LaFreniere 
Michael  Cronin 

703-751-7290 

703-751-7290 

703-751-7290 

703-751-7290 

morsone@aol.com 

naddison@msis.dmso.mil 

clafreni@msis.dmso.mil 

mcronin@tecnetl.jcte,jcs.mil 

by  sending  this  electronically  to  a  group  of  to  include  you.  Also,  the  preliminary  pro-  http://www.msosa.mil/groups/mors/.  In 

you  to  see  how  this  will  work.  If  you  are  gram  that  you  will  get  in  January  will  not  future  issues  of  PHALANX,  we  will  discuss 

interested  in  participating  in  this  test  of  duplicate  any  of  the  material  in  the  call  for  the  program  schedule  in  greater  detail, 

electronic  registration,  please  send  an  e-  papers.  This  should  simplify  the  commu-  So,  mark  the  calendar:  June  10-12, 

mail  to  the  MORS  office  nications  process.  All  of  the  materials  will  1997.  And,  begin  to  plan  on  how  you  will 
(morsoffice@aol.com),  and  we  will  strive  be  made  available  on  our  Web  page  at  contribute  to  a  great  annual  symposium.  O 
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MORS  AWARDS 


Brady  Wins  Wanner  Award 

Michael  Cronin 


Schnauppner  Wins 
Tisdale  Award 


Edward  Brady,  FS  was  named  the 
1996  recipient  of  the  Vance  R.  Wanner 
Award,  the  Society’s  most  prestigious  acco¬ 
lade.  The  award  was  formally  presented  at 
the  plenary  session  of  the  64th  MORS 
Symposium  at  Fort  Leavenworth,  KS. 

The  award  is  presented  annually  to  a 
person  who  best  represents  the  ideals  of  and 
has  made  significant  contributions  to  the 
operations  research  community  and  the  Mil¬ 
itary  Operations  Research  Society.  It  is 
named  to  honor  the  memory  of  MORS’  first 
Executive  Secretary. 

During  his  career,  Brady  has  contributed 
significantly  to  the  profession  of  military 
operations  research  and  to  MORS.  For  the 
profession  he  has  served  or  is  serving  on 
numerous  advisory  boards  ranging  from 
studies  to  intelligence  in  addition  to  having 
been  the  MITRE  Corporations’  Washington 
Group  President.  For  MORS,  he  served  as  a 
MORS  President,  chaired  three  workshops, 
and  is  the  only  individual  to  have  chaired 
two  annual  symposia.  His  dedication  and 
demonstrated  capabilities  clearly  make  him 
a  deserving  recipient  of  this  prestigious 
award. 

Following  are  Brady’s  comments  upon 
receiving  this  award. 

“I  was  very  surprised  to  be  nominated 
for  and  receive  the  Wanner  Award.  Such  an 
honor  was  the  furthest  thing  from  my  mind. 
I  very  much  appreciate  and  feel  strongly 
that  nothing  is  as  personally  rewarding  as 
being  recognized  by  your  professional 
peers. 

“A  little  over  thirty  years  ago  when  I 
first  studied  Morse  and  Kimball  at  the 
Naval  Academy,  I  had  no  idea  that  through¬ 
out  my  lifetime  I  would  have  repeated  inter¬ 
esting,  challenging  and  rewarding  experi¬ 
ences  in  conjunction  with  the  analysis  of 
military  operations.  From  my  perspective, 
the  current  era  offers  these  same  opportuni¬ 
ties  to  all  of  you  working  in  this  field.  The 
end  of  the  Cold  War  and  the  budgetary  dri¬ 
ven  reductions  in  the  size  of  our  military 
forces  has  caused  many  important  questions 
to  be  asked  by  leading  decision  makers. 
Many  of  these  questions  are  very  demand¬ 
ing  in  terms  of  their  need  for  clarity  of 


thought,  openmindness,  and  attention  to 
appropriate  use  of  methodologies  and  ana¬ 
lytical  tools.  Some  illustrations  are:  consid¬ 
eration  of  the  likely  role  and  use  of  the  mili¬ 
tary  in  the  world  over  the  next  twenty-five 
years,  the  possibility  of  radically  changing 
operational  concepts  and  use  of  forces,  the 
net  value  to  the  nation  of  one  type  of  force 
or  weapon  compared  to  another,  how  to 
analyze  military  operations  which  don’t 
involve  combat  and  attrition,  etc. 

“Today’s  Defense  establishment  is  pur¬ 
suing  many  interesting  approaches  to  gener¬ 
ate  the  needed  answers,  although  some  con¬ 
tinue  to  live  in  the  intellectual  past  and  resist 
looking  at  our  future.  These  approaches  fre¬ 
quently  utilize  warfighting  experiments, 
experimental  testbeds,  and  distributed  simu¬ 
lation  sometimes  including  large  human-in- 
the-loop  exercises.  This  is  a  challenging 
arena  for  operations  analysts.  To  my  mind, 
our  community  should  be  taking  a  leading 
role  in  these  areas,  but  I  strongly  fear  that 
we  are  not.  Experiments,  exercises,  and  the 
use  of  testbeds  to  support  decisions  require 
validity  and  analytical  structure  to  be  valu¬ 
able  in  the  long  run.  It  is  simply  not  suffi¬ 
cient  to  witness  a  few  events  and  decide  that 
“it  looks  good  to  me.”  We  must  make  our 
voices  heard  in  these  arenas,  and  we  can 
only  do  so  by  direct  participation.  I  worry 
that  many  of  us  are  not  doing  so  but  rather 
are  sitting  on  the  sidelines  saying  “I’m  sorry 
I  don’t  have  an  equation  or  a  simulation  that 
fits  that  question.”  The  end  result  of  such  an 
approach  can  only  be  for  decision  makers  to 
conclude  correctly  that  we  are  no  longer  rel¬ 
evant  to  their  problems. 

“A  brief  word  about  MORS  itself  Over 
the  past  twelve  years,  MORS  has  been  like 
an  extended  professional  family  for  me,  and 
it  can  be  the  same  for  you.  I  have  learned 
many  things,  and  met  and  been  befriended 
by  many  fine  people  who  are  important  to 
me  personally,  as  well  as  being  a  valuable 
resource  for  me.  To  have  this  as  an  outcome 
requires  more  than  just  coming  to  MORS, 
giving  a  paper  and  immediately  leaving.  It 
takes  a  commitment  of  time  and  effort  to 
participate  fully,  and  thereby  to  benefit.  I 
urge  you  to  do  so.”0 


Michael  Cronin 


LT  Schauppner  receives  the  Tisdale 
Award  from  RADM  Marsha  J.  Evans  dur¬ 
ing  the  March  graduation  exercises. 


A  committee  of  Naval  Postgraduate 
School  Operations  Research 
Department  Faculty  selected  Lieu¬ 
tenant  Craig  T.  Schnauppner  as  the 
March  1996  recipient  of  the  Stephen  A. 
Tisdale  Graduate  Research  Prize.  LT 
Schnauppner  earned  the  award  for  his  the¬ 
sis  titled  “An  Aircraft  Carrier  Deployment 
Scheduling  Model.”  The  prize,  presented 
for  the  best  thesis  in  each  graduating  class 
at  NPS,  honors  the  memory  of  LCDR 
Stephen  A.  Tisdale,  who  was  killed  in  a 
mid-air  collision  in  1991. 

LT  Schauppner  graduated  from  the  Uni¬ 
versity  of  California  Los  Angeles  in  1988 
with  a  B.S.  in  Physics.  He  received  his 
commission  as  an  Ensign  in  the  U.S.  Navy 
from  the  NROTC  program  at  UCLA.  Pre¬ 
vious  assignments  have  included  VT-2  and 
VT-31  at  NAS  Whiting  Field,  FL,  and  VP- 
31  and  VP-9  at  NAS  Moffet  Field,  CA.  He 
received  a  Naval  Achievement  Medal  for 
superior  performance  of  duty  while 
attached  to  VP-9.  The  text  of  the  abstract  of 
his  Thesis  follows. 
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Abstract 

The  Navy’s  peacetime  mission  is  “to 
conduct  forward  presence  operations  to 
help  shape  the  strategic  environment  by 
deterring  conflict,  building  interoperability, 
and  by  responding,  as  necessary,  to  fast 
breaking  crises  with  the  demonstration  and 
application  of  credible  combat  power.”  To 
meet  this  mission,  the  Navy  deploys  air¬ 
craft  carriers  to  forward  positions  through¬ 
out  the  world.  A  new  nuclear  powered  air¬ 
craft  carrier  costs  over  3.4  billion  dollars 
and  when  deployed  carries  over  6,000  per¬ 
sonnel  on  board.  Considering  the  cost  and 
man-hours  involved  in  carrier  operations, 
judicious  and  effective  use  of  these  valu¬ 
able  assets  is  imperative. 

The  CINCPACFLT  Operations  Depart- 


The  United  States  Military  Academy 
Departments  of  Systems  Engineer¬ 
ing  and  Math  Sciences  announce 
the  winners  of  the  1996  Hollis  Award  for 
excellence  in  Operations  Research: 

Cadet  IMatthew  P.  Champion 
Cadet  Jonathan  P.  Darr 
Cadet  Paul  J.  Ferrigno 
Cadet  Hernan  E.  Ruiz 
Cadet  Kyle  W.  Towns 

Jointly  sponsored  by  the  Academy’s 
Operations  Research  Center  and  Mathe¬ 
matical  Sciences  Center  of  Excellence,  the 
Hollis  Award  is  presented  annually  to  indi¬ 
vidual  cadets,  cadet  teams,  or  cadet/faculty 
teams  at  the  Academy  in  recognition  of 
excellence  in  military  operations 
research/systems  analysis.  The  award  is 
presented  for  military  OR/SA  work  that 
best  contributes  to  enriching  cadet  educa¬ 
tion,  enhancing  professional  development, 
and  linking  the  Academy  to  the  Army  by 
addressing  issues  confronting  the  profes¬ 
sion  of  arms. 

The  award  recognizes  the  lifetime  of 
selfless  service  of  Mr.  Walter  W.  Hollis, 
FS  to  the  Army  and  nation.  Mr.  Hollis  cur¬ 
rently  serves  as  the  Deputy  Under  Secre¬ 
tary  of  the  Army  (Operations  Research),  a 


ment  maintains  a  five  year  deployment  plan 
for  the  six  carriers  assigned  to  the  Pacific 
Fleet.  Currently,  the  deployment  schedule 
is  produced  manually.  A  feasible  five  year 
plan  typically  takes  the  carrier  scheduling 
officer  one  week  to  generate.  This  thesis 
presents  an  optimization  based  tool  to  assist 
in  constructing  deployment  schedules  that 
maximize  the  forward  presence  of  Pacific 
Fleet  carriers.  The  underlying  optimization 
model  is  different  from  those  in  the  litera¬ 
ture.  Instead  of  using  a  set  covering 
approach,  the  problem  is  formulated  as  a 
shortest  path  problem  with  side  constraints. 
This  formulation  allows  the  problem  to  be 
solved  more  rapidly,  thus  allowing  more 
possibility  for  sensitivity  and  trade-off 
analyses.  O 


position  he  has  held  since  1980. 

This  year’s  winning  team  designed  and 
built  a  computer  simulation  tool  that 
enables  a  user  to  evaluate  the  effectiveness 
of  different  minefield  configurations.  The 
user  designs  a  minefield  pattern  by  placing 
mines  on  the  screen  using  a  mouse  and 
associated  icons.  The  evaluation  program 
simulates  vehicles  entering  the  minefield 
from  different  locations  and  calculates 
appropriate  statistics  which  can  be  used  to 
evaluate  and  optimize  the  effectiveness  of 
the  minefield.  The  cadet  team  used  this 
tool  to  analyze  and  compare  several  alter¬ 
native  minefield  designs  of  interest  to  the 
project’s  sponsor,  the  US  Army  Armament 
Research,  Development  and  Engineering 
Center  (ARDEC),  Picatinny,  New  Jersey. 
The  project  was  directed  by  ETC  John  A. 
Marin,  Department  of  Systems  Engineer¬ 
ing. 

The  award  was  presented  30  May  in  the 
Thayer  Awards  Room.  Award  winners 
received  a  plaque,  a  $50.00  cash  prize,  a 
letter  of  commendation  from  the  Superin¬ 
tendent  and  the  opportunity  to  present  their 
work  at  the  64^^  Military  Operations 
Research  Symposium.  A  permanent 
plaque  engraved  with  the  names  of  the 
recipients  will  be  displayed  in  the  Depart¬ 
ment  of  Systems  Engineering.  O 
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tify  their  needs  for  advanced  M&S.  While 
several  communities  embraced  this  oppor¬ 
tunity  (e.g.,  training,  T&E),  the  analysis 
community  demurred.  The  Synthesis 
Panel  further  observed  that  there  is  no  long 
range  vision  to  guide  the  allocation  of  lim¬ 
ited  M&S  development  resources  to  sup¬ 
port  the  analyst.  Consequently,  the  analy¬ 
sis  community  tends  to  be  excessively 
reactive  to  near-term  needs  at  the  expense 
of  needed  longer-term  investments. 

The  Synthesis  Panel  found  that  the 
absence  of  an  “ADS  Plan  for  the  Analysis 
Community”  has  resulted  in  fragmented 
action  within  the  analysis  community  and 
limited  the  leveraging  of  the  resources  and 
expertise  that  are  resident  in  other  commu¬ 
nities  (e.g.,  training,  acquisition).  To  ame¬ 
liorate  those  shortfalls,  the  Analysis  Coun¬ 
cil  that  has  been  formed  under  the 
Executive  Committee  for  M&S  (EXCIMS) 
must  take  decisive  action.  It  should  devel¬ 
op  the  vision  for  the  next  generation  of 
M&S  tools  for  the  analyst  and  draft  an 
associated  “ADS  Plan  for  the  Analysis 
Community.” 

Community  Education,  Training  & 
Experience.  The  Synthesis  Panel  found 
that  a  broad  set  of  skills,  knowledge,  and 
expertise  will  be  needed  by  the  community 
if  it  is  to  be  able  to  appreciate  the  capabili¬ 
ties  and  limitations  of  ADS  and  to  employ 
ADS  effectively  and  efficiently.  It  must  be 
emphasized  that  the  community  in  ques¬ 
tion  subsumes  both  analysts  and  decision 
makers.  In  the  absence  of  appropriate  edu¬ 
cation,  training,  and  experience,  the  results 
that  analysts  derive  from  the  application  of 
ADS  are  likely  to  be  suspect  and  the  deci¬ 
sion  makers’  understanding  of  the  meaning 
and  validity  of  those  results  is  likely  to  be 
limited. 

To  deal  with  these  issues,  the  Synthesis 
Panel  put  forth  several  recommendations. 
First,  key  Service  schools  should  develop 
curricula  and  offer  a  sub-specialty  of  “ADS 
Analyst”.  Candidate  schools  include  the 
Naval  Postgraduate  School  (NPS),  the  Air 
Force  Institute  of  Technology  (AFTT),  and 
the  Army  Logistics  Management  College 
(ALMC).  Second,  assignments  to  “ADS 
Analyst”  positions  should  be  of  suitable 
duration  and  consistency  to  ensure  that 
necessary  experience  is  acquired  and  sus- 

(See  ADSA  ’96,/?.  22} 
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mined.  Thus,  for  example,  once  an  analyst 
has  qualified  as  an  “ADS  Analyst”  at  one 
of  the  above  institutions,  his/her  next 
assignment  should  be  to  an  ADS  activity  or 
to  an  organization  which  may  perform  col¬ 
laborative  analysis  employing  ADS. 
Third,  in  support  of  decision  maker  educa¬ 
tion  and  training,  appropriate  programs  on 
ADS  should  be  developed  and  offered  at 
institutions  such  as  the  Defense  Systems 
Management  College  (DSMC),  the 
Defense  Acquisition  University  (DAU), 
the  National  Defense  University  (NDU), 
and  Senior  Defense  Service  Schools. 

Qualitative  assessment  of  the  Work¬ 
shop.  To  establish  a  qualitative  measure  of 


the  effectiveness  of  the  Workshop,  the 
Synthesis  Panel  informally  assessed  the 
Working  Groups  against  the  four  ToR 
objectives,  based  on  its  observations  of 
their  operations  and  products.  These 
assessments  were  in  the  form  of  a  color 
(i.e.,  “green”  implied  fairly  complete 
achievement  of  the  objective,  “amber” 
implied  moderate  achievement  of  the 
objective,  and  “red”  implied  a  failure  to 
achieve  the  objective). 

•  The  first  objective  was  to  assess  the  utili¬ 
ty  of  current  and  future  (3-5  years  out) 
ADS-based  capabilities  for  analytic 
applications,  especially  possible  areas  of 
new  analytic  capabilities.  The  Synthesis 
Panel  rated  this  as  “amber.” 

•  The  second  objective  was  to  define  areas 


INTELLIGENCE 

(continued  from  p.  17) 

Third,  TPO  has  initiated  the  WARSIM 
Intelligence  Module  (WIM)  project  to 
develop  the  follow-on  system  to  TACSIM. 
WIM  is  an  integral  component  of  the 
Army’s  Warfighters  Simulation  (WAR¬ 
SIM)  2000,  and  will  consequently  also  be 
in  the  Joint  Simulation  System  (JSIMS). 
The  architecture  issues  discussed  above 
will  be  addressed  within  those  develop¬ 
ment  programs. 

Summary 

TACSIM,  NWARS,  SIGS,  HRSS,  JSS, 
and  MUSE  have  collaboratively  evolved  a 
system  to  support  training  of  Army  and 
joint  commanders  and  staffs.  This  system 
of  intelligence  simulations  is  complex  and 
requires  constant  maintenance. 

The  system  has  and  will  continue  to 
effectively  support  numerous  exercises 
annually  including  the  Army’s  Prairie 
Warrior,  U.S.  Atlantic  Command’s  Unified 
Endeavor  series,  U.S.  Forces  Korea’s 
Ulchi  Focus  Lens,  among  others. 

The  challenge  over  the  next  several 
years  is  to  ensure  a  successful  transition 
from  the  current  generation  to  the  next 
generation  of  simulations  developed  under 
the  WARSIM  and  JSIMS  environments. 
To  meet  these  challenges,  TPO  will  main¬ 
tain  the  current  TACSIM  and  TACSIM 
interfaces  to  other  intelligence  simulations, 
make  modest  improvements  to  ensure  the 


maintainability  of  the  current  system  until 
the  next  generation  of  simulations  is  field¬ 
ed,  and  participate  in  the  development  of 
the  next  generation  of  simulations. 
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MAJ  Johnny  Bullington  is  the  Project 
Director  for  the  US  Army  Tactical  Simula¬ 
tion  (TACSIM)  Project  Office  of  the  US 
Army  Simulation,  Training,  and  Instrumen¬ 
tation  Command  (STRICOM).  Major 
Bullington  has  had  previous  assignments 
as  a  Ground  Surveillance  Radar  Platoon 
Leader  and  Adjutant/Sl  with  the  533rd  Ml 
Bn  (CEWI)  in  the  3rd  Armored  Division; 
Assistant  Battalion  S3  and  HHC  Company 


of  needed  enhancements  of  ADS-based 
capabilities  to  increase  their  utility  to  the 
military  analytic  community.  The  Syn¬ 
thesis  Panel  rated  this  as  “amber.” 

•  The  third  objective  was  to  increase  the 
military  analytic  community’s  under¬ 
standing  of  current  ADS-based  capabili¬ 
ties  and  near-term  improvement  pro¬ 
grams.  The  Synthesis  Panel  rated  this  as 
“green”. 

•  The  fourth  objective  was  to  increase 
interactions  between  the  military  analytic 
community  and  developers  of  ADS- 
based  capabilities.  The  Synthesis  Panel 
rated  this  as  “green”.  O 


commander  in  the  202nd  Ml  Battalion; 
Collection  Manager  and  Chief,  Production 
in  the  513th  Military  Intelligence  Brigade; 
and  Threat  System  Acquisition  Manager  at 
the  US  Army  Combined  Arms  Center. 
Major  Bullington  is  a  graduate  of  the  Unit¬ 
ed  States  Army  Command  and  General 
Staff  College  and  holds  a  Master  of  Busi¬ 
ness  Administration  from  the  University  of 
Tennessee. 

Mr,  Gordon  Miller  is  a  Member  of  the 
Technical  Staff  in  the  Information  Systems 
and  Technologies  Division  of  The  MITRE 
Corporation  and  provides  systems  engi¬ 
neering  support  to  the  TACSIM  Project 
Office.  Before  joining  MITRE,  he  was  a 
Senior  Analyst  at  Science  Applications 
International  Corporation  and  worked  on 
a  variety  of  modeling  and  simulation  and 
program  acquisition  projects.  Before  retir¬ 
ing  from  the  US  Army,  he  had  assignments 
at  the  HQDA  staff,  US  Army  Concepts 
Analysis  Agency,  Central  Intelligence 
Agency  and  the  US  Army  Chemical  School 
as  well  as  numerous  tactical  units.  He 
received  a  Bachelors  of  Science  in  Chem¬ 
istry  from  Worcester  Polytechnic  Institute 
and  a  Master  of  Science  in  Business 
Administration  from  Boston  University.  O 
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OR  AT  THE  SERVICE  ACADEMIES 


Forecasting  Issues  for  the  Soldier  of  the  Future 


Military 
operations 
research 
(OR)  has  a  rich  histo¬ 
ry  of  forecasting  or 
predicting  how  the 
warfighter  will  per¬ 
form  in  combat,  from 
the  simple  back-of- 
the-envelope  model 
to  the  million-lines- 
of-code  simulation 
distributed  and  linked  to  other  simulations, 
models  and  live  military  exercises.  Fine 
tuning  these  models  and  simulations  has 
been  the  job  of  major  military  organiza¬ 
tions,  such  as  TRAC,  CAA  or  the  schools 
and  academies. 

Decades  of  highly  trained  officers  have 
come  from  these  schools  and  academies  to 
tackle  the  applied  problems  of  military 
combat  and  operations  other  than  war. 
Since  the  late  1960s,  these  computerized 
models  and  simulations  were  for  the  most 
part  quantitative  tools  and  closed  systems. 
Recently,  more  open  systems  are  begin¬ 
ning  to  emerge  as  well  as  qualitative  fac¬ 
tors  beginning  to  reemerge  into  combat 
and  training  simulations,  which  is  con¬ 
tributing  to  a  more  robust  set  of  OR  tools. 

However,  one  area  that  may  be  as 
important  is  the  area  of  the  econometric 
forecasting  model  to  answer  some  really 
tough  questions.  Recently,  many  of  these 
questions  were,  and  are,  about  forecasting 
technology  trends  or  alternative  worlds  for 
the  timeframe  of  Force  XXL  Adaptive 
computer  modeling  algorithms,  better  data 
access  methods,  and  more  involvement 
from  senior  military  leaders  has  helped 
improve  such  forecasts.  But,  one  focus 
area  still  needs  addressing.  That  is  in  the 
area  of  forecasting  the  human  element  not 
in  the  combat  model  or  simulation,  but  in 
the  real  world  of  recruitment. 

To  many,  econometric  models  of 
recruitment  supply  and  demand  may  not 
seem  as  challenging  as  developing  a  joint 
service  distributed  confederation  of  simula¬ 
tions.  But,  consider  how  well  you  know 
that  soldier  and  the  key  assumptions  that 
describe  the  soldier’s  behavior  on  the  bat¬ 
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tlefield.  The  M&S  community  is  just 
beginning  to  understand  the  high-low  mix 
of  soldiers  similar  to  the  high-low  mix  of 
weapons  used  in  a  model  or  simulation. 
But,  what  does  not  seem  clear  is  that  high- 
low  quality  of  that  soldier,  not  for  this  year, 
but  for  the  years  of  Force  XXI,  when  many 
of  you  will  not  be  in  active  duty  status  or 
may  be  retired  civilians. 

For  that  soldier  of  Force  XXI,  what  do 
the  officers  and  enlisted  need  to  know  in 
order  to  do  that  task,  to  carry  out  their  mis¬ 
sion,  to  survive?  How  does  the  military 
take  advantage  of  the  information  technol¬ 
ogy  to  train  these  soldiers  twenty  years  into 
the  future?  These  questions  may  not  be  as 
urgent  an  issue  for  the  would-be  officers  as 
it  may  be  with  the  enlisted.  So,  the  focus 
shifts  to  that  high  school  student  who  grad¬ 
uates  in  2020  and  decides  (somehow)  to 
join  the  military  service.  What  perfor¬ 
mance  levels  will  be  needed  of  that  green 
soldier?  Will  that  high  school  student  be 
able  to  meet  those  needs,  the  military  stan¬ 
dards?  In  examining  census  data,  there 
should  be  enough  supply  of  high  school 
graduates,  but  recruiting  data  show  a  trend 
that  the  demand  is  not  or  may  not  meet  the 
supply.  How  to  address  these  supply  and 
demand  side  algorithms  depends  on  the 
quality  of  the  data  and  the  assumptions  the 
analyst  brings  to  his  data  and  models, 

COL  Chuck  Kaylor,  US  Army 
Recruiting  Command,  indicates  that  the 
forecasting  models  for  the  short  and  long 
term  seem  reasonable.  What  is  missing  is 
a  set  of  mid-range  forecasting  tools. 
RAND  is  also  working  with  OSD  and  the 
Army  on  this  issue  and  is  continuing  to 
examine  economic  trends  that  may  affect 
the  short,  mid  and  long-range  modeling 
and  analysis  efforts. 

For  these  econometric  models,  one 
problem  is  data.  What  data  is  usable? 
What  are  the  key  assumptions  about  the 
data  that  are  usable?  Are  data  from  the 
pre-Cold  War  useful  in  a  post-Cold  War 
world?  How  do  you  handle  time-series 
data?  Finding  a  statistically  significant 
answer  may  require  going  beyond  tradi¬ 
tional  statistical  methods,  such  as  logistics 
regression  into  the  world  of  neural  net¬ 


works.  The  independent  variable  for  these 
econometric  models  is  usually  the  number 
of  high-quality  students  or  recruits  coming 
from  the  high  school  environment.  The 
dependent  variables  could  be  civilian  and 
military  pay,  employment  trends,  popula¬ 
tion  size  in  general  or  within  a  recruiting 
area,  race,  gender,  test  scores,  military 
advertising  costs  and  various  benefits. 

The  service  academies  are  an  elite  intel¬ 
lectual  breeding  ground  for  innovations 
and  ideas  for  alternative  analytical  solu¬ 
tions  to  today’s  complex  military  prob¬ 
lems,  beyond  those  of  the  combat  model 
and  simulation.  What  we  will  explore  in 
the  future  issues  are  some  of  these  forecast¬ 
ing  modeling  aspects  and  key  analytical 
challenges  to  the  OR  community.  In  the 
meantime,  if  you  have  been  working  on 
econometric  forecasting  models,  or  ethno¬ 
graphic  data  collections,  or  related  aspects 
to  forecasting  human  behavior,  we  would 
welcome  your  contribution.  Send  any  pro¬ 
posals,  abstracts,  model  information,  ana¬ 
lytic  results,  etc.  to  email: 
hedgewo@dcsopspo3.army.mil  or  facsimi¬ 
le  to  Dr.  Oliver  Hedgepeth  at  (703)  695- 
9879. 


/ 


Some  suggested  readings  (courtesy  of 
Bruce  Orvis,  RAND)  to  acquaint  you  with 
this  issue  are: 
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Asch,  B.J.,  “Navy  Recruiter  Productivity 
and  the  Freeman  Plan,”  R-3713-FMP, 
RAND,  June  1990. 

Asch,  B.J,  and  B.R.  Orvis,  “Recent 
Recruiting  Trends  and  Their  Implica¬ 
tions:  Preliminary  Analysis  and  Recom¬ 
mendations,”  MR-549-A/OSD,  RAND, 
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Cotterman,  R.F,,  “Forecasting  Enlistment 
Supply:  A  Time-Series  of  Cross-Sec¬ 
tions  Model,”  R-3252-FMP,  RAND, 
July  1986.  nO^:7-(^ 

Orvis,  B.R.,  N.  Sastry  and  L.L.  McDonald, 
“Recent  Recruiting  Trends  and  Their 
Implications:  Interim  Report,”  DRR- 
1 175-A/OSD,  RAND,  August  1995. 

^  Working,  E.,  “What  Do  Statistical  Demand 
Curves  Show?,”  Quarterly  Journal  of 
Economics,  Vol.  41,  1926,  pp.  212-235. 
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Openers 

Robert  Eberth 

Palmer-Eberth 

When  Dr.  Julian  Palmore,  the 

new  Editor  of  PHALANX, 
asked  me  to  take  on  the  role  as 
department  editor  of  Technical  Forum,  we 
found  ourselves  in  immediate  and  enthusi¬ 
astic  agreement  on  what  the  column  should 
be  and  do.  Specifically,  the  overall  pur¬ 
pose  of  the  column  will  be  to  identify  and 
illuminate  critical  technical  issues  of  today 
in  Operations  Research  in  general  and 
Modeling  and  Simulation  in  particular. 

‘Technical”  does  not  necessarily  mean 
“technology.”  Many  of  the  issues  are 
expected  to  deal  with  the  theoretic  founda¬ 
tion  tifciat  in  fact  underlies  all  that  our  tech¬ 
nologies  implement. 

The  goal  may  be  rather  lofty:  to  stimu¬ 
late  advances  in  the  state  of  the  art,  and 
with  that  “ait”  being  very  broadly  defined. 

The  objectives  are  to  introduce,  define, 
and  explore  critical  issues.  The  column  is 
intended  to  challenge  the  MORS  commu¬ 
nity  to  look  critically  at  —  and  engage  in  a 
constructive  dialogue  over  —  some  of  our 
most  cherished  paradigms  as  well  as  at 
some  new  or  untried  alternative  technical 
approaches. 

For  openers,  I  want  to  look  at  two 
intertwined  issues.  The  first  is  the  highly 
popular  but  increasingly  one-sided  debate 
about  the  “right”  kind  of  simulation 
approach.  The  second  is  the  possible  dis¬ 
sonance  in  our  pursuit  of  detail,  fidelity, 
realism,  and  accuracy. 

lype  of  Simulation 

First,  is  there  a  single  type  of  simulation 
approach  that  is  inherently  superior  for  par¬ 
ticular  application  environments  or  for  the 
new  generation  of  major  simulations  as  a 
whole?  The  “new  generation”  I  have  in 
mind  comprises  the  Joint  Simulation  Sys¬ 
tem  (JSIMS),  initially  focused  on  training; 
the  Joint  Warfare  Simulation  (JWARS), 
initially  focused  on  analysis  and  assess¬ 
ment;  and  the  Joint  Modeling  and  Simula¬ 
tion  System  (J-MASS),  initially  focused  on 
acquisition  and  Test  and  Evaluation 
(T&E). 

The  debate  to  now  has  centered  around 


two  types:  (1)  aggregate-level  or  “mass- 
action”  simulations,  and  (2)  entity-level 
simulations  that  are  “scaled  up”  to  look  at 
results  at  higher  levels  of  force  aggrega¬ 
tion.  There  are  other  options  that  are  not 
covered  by  those  two  broad  categories,  and 
some  of  the  “other”  options  will  be  men¬ 
tioned  below,  but  those  two  categories 
dominate  the  debate  today. 

Aggregate-Level  Simulations 

From  one  perspective,  aggregate-level 
simulations  can  be  viewed  as  dinosaurs. 
That  is,  they  were  absolutely  necessary 
when  computer  processing  speeds  and 
capacities  could  not  handle  the  extremely 
high  entity  counts  and  associated  computa¬ 
tional  loads  that  would  be  required  to  scale 
up  from  individual  entities  to  theater-level 
conflicts,  even  with  “low  fidelity”  entities 
and  deterministic  interactions. 

IJiat  situation  has  changed  dramatical¬ 
ly  with  the  capabilities  of  the  computer 
today,  especially  if  we  look  at  the  capaci¬ 
ties  and  speeds  of  the  massively  parallel 
processors.  In  short,  we  may  no  longer 
have  to  use  aggregate-level  simulations  to 
model  campaign  or  even  theater-level  war¬ 
fare,  although  large  “high  fidelity”  entity 
counts  coupled  with  configural  considera¬ 
tions  and  stochastic  interactions  could  fully 
load  even  modem  computers. 

Still,  aggregate-level  simulations  can 
be  highly  useful  in  their  own  right.  They 
can  engender  a  great  deal  of  insight  into  a 
particular  decision  environment  in  a  rela¬ 
tively  short  period  of  time.  They  have 
proven  highly  useful  for  analysis  “at  the 
margin.”  That  is,  sensitivity  and  contin¬ 
gency  analyses  are  employed  with  the 
aggregate-level  simulation  to  determine  the 
relative  utility  of  some  operational  concept 
or  capability. 

Their  particular  advantage,  relative  to 
entity-level  simulations,  is  that  they  are 
able  to  very  rapidly  perform  the  iterations 
necessary  to  those  sensitivity  and  contin¬ 
gency  analyses. 

Along  with  that,  they  have  the  addi¬ 
tional  advantage  of  being  “logically  trans¬ 


parent”  (assuming  reasonable  documenta¬ 
tion).  If  the  analyst  using  the  simulation 
sees  something  counterintuitive  develop¬ 
ing,  it  is  relatively  easy  to  determine 
whether  it  is  the  result  of  some  artificiality 
within  the  simulation,  the  result  of  an  error 
in  the  simulation  or  the  data,  or  a  new 
insight  into  the  combat  environment  or 
process. 

But,  by  attempting  to  capture  the 
processes  of  combat  (movement,  attrition, 
etc.)  through  deterministic  algorithms 
operating  on  homogeneous  masses  of 
forces,  such  simulations  give  up  the  ability 
to  see  and  work  with  potentially  critical 
entity-level  capabilities  and  interactions. 
The  possibly  unique  contributions  —  both 
positive  and  negative  —  of  single  entities 
to  the  course  or  outcome  of  combat  are 
“averaged  out”  by  mass-action  models. 

Also,  structural  artificialities  within 
aggregate-level  simulations  can  mean  that 
the  abilities  of  the  analyst  become  the  sin¬ 
gle  most  critical  aspect  to  achieving  valid 
results.  As  an  example  of  potential  prob¬ 
lems  at  the  working  level,  in  aggregate- 
level  simulations  the  order  of  processing  of 
actions  can  take  on  a  crucial  and  wholly 
artificial  importance  with  regard  to 
absolute  numeric  outcomes  (e.g.,  losses  to 
covering  fires  may  be^  computed  prior  to 
losses  to  mines,  giving  mines  a  structural 
disadvantage  in  terms  of  their  apparent 
effectiveness  as  casualty-inflicting 
weapons). 

Similarly,  aggregate-level  simulations 
use  a  wide  variety  of  stylized  concepts  of 
massed  position  and  movement  (e.g.,  par¬ 
allel  “pistons”  representing  land  force 
movement)  that  can  inject  discontinuities 
across  artificial  geographic  boundaries.  If 
the  analyst  does  not  recognize  and  account 
for  those  types  of  structural  effects,  results 
can  be  wholly  misleading. 

Finally,  aggregate-level  simulations 
could  well  be  held  to  be  irrelevant  for  any¬ 
thing  other  than  attrition  warfare.  In  partic¬ 
ular,  it  is  difficult  to  conceive  their  applica¬ 
tion  in  modeling  Operations  Other  Than 
War  (OOTW),  where  the  usefulness  of 
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attrition  as  a  measure  of  effectiveness 
would  be  nonexistent. 

The  same  type  of  concern  applies  to 
their  use  in  modeling  maneuver  warfare. 
And,  of  course,  the  ultimate  condemnation 
of  today:  aggregate-level  simulations  don’t 
look,  feel,  or  act  “realistic.” 

Entity-Level  Simulations 

Does  the  above  discussion  mean  that 
entity-level  simulations  are  then  the  “cor¬ 
rect”  choice  virtually  by  default?  Certainly 
their  entities  can  be  made  to  look  real  (par¬ 
ticularly  if  a  system  were  to  use  the 
“immersive  displays”  being  marketed 
today  for  simulations),  and  can  be  made  to 
explicitly  represent  all  the  functions  of 
their  real-world  counterparts. 

The  potential  advantages  for  that  level 
of  detail  —  for  at  least  the  training  environ¬ 
ment  —  have  to  be  acknowledged.  But 
does  it  do  anything  for  the  analytic  envi¬ 
ronment?  For  the  acquisition  or  T&E  envi¬ 
ronments? 

Detail,  fidelity,  realism,  accuracy 

Entity-level  modeling  has  an  almost 
natural  basis  of  support  because  it  lends 
itself  directly  to  the  never-ending  search 
for  greater  detail  in  our  simulations. 

It’s  an  article  of  faith:  greater  detail 
provides  greater  fidelity.  In  fact,  the  two 
have  become  virtually  synonymous,  both 
with  each  other  and  with  “realism.” 

The  draft  Glossary  of  Modeling  and 
Simulation  Terms  of  the  Defense  Model¬ 
ing  and  Simulation  Office  (DMSO)  recog¬ 
nizes  three  definitions  for  “fidelity:” 

•  “The  similarity,  both  physical  and 
functional,  between  the  simulation 
and  that  which  it  simulates.” 

•  “A  measure  of  the  realism  of  the 
simulation.” 

•  “The  degree  to  which  the  represen¬ 
tation  within  the  simulation  is  simi¬ 
lar  to  a  real  world  object,  feature  or 
condition  in  a  measurable  or  per¬ 
ceivable  manner.” 

The  24  May  1996  draft  of  the  Opera¬ 
tional  Requirements  Document  for  the 
Joint  Modeling  and  Simulation  System  (J- 
MASS)  also  reinforces  the  conceptual  tie 
between  fidelity  and  detail: 

“[J-MASS  shall]  Provide  for  multi¬ 
level  fidelity/resolution  capability. 

Fidelity  refers  to  the  level  of  detail  J- 

MASS  and  J-MASS  models  must  rep¬ 


resent.”  [emphasis  added] 

But,  is  detail  for  its  own  sake  what  we 
are  really  pursuing?  What  are  we  striving 
for  when  we  talk  about  fidelity  or  realism? 
Isn”t  it  accuracy  —  specifically,  the  accu¬ 
racy  of  predictions  of  performance  and 
results  —  that  we  actually  are  after? 

Certainly,  for  simulations  that  are 
intended  to  support  analysis,  or  acquisition 
decisions,  or  T&E,  predictive  accuracy  is 
the  critical  attribute  of  a  simulation.  That 
same  predictive  accuracy  may  take  a  back 
seat  to  perceptual  realism  in  some  narrow¬ 
ly-defined  (stimulus-response)  training 
applications,  but  will  be  of  some  concern 
even  there. 

If  we  accept  that  we  are  in  fact  in  pur¬ 
suit  of  greater  predictive  accuracy,  then  the 
problem  for  this  discussion  becomes  one  of 
a  challenge  to  the  article  of  faith  mentioned 
earlier:  does  greater  detail  at  the  entity 
level  imbue  greater  predictive  accuracy  to 
the  simulation  as  a  whole? 

The  question  seems  trivial.  How  could 
greater  detail  not  promote  greater  accura¬ 
cy?  The  answer  is  that  the  two  exist  and 
have  to  be  addressed  in  different  frames  of 
reference.  The  detail  that  we  inject  will  be 
in  the  computer.  The  accuracy  that  we 
seek  is  in  the  real  world.  And,  the  laws  of 
probability  work  against  us  when  we  tran¬ 
sition  between  those  two  frames. 

For  the  moment,  disregard  presenta¬ 
tional  detail  (how  real  the  simulation  looks 
or  —  for  virtual  simulations  —  how  real  it 
feels  to  the  simulation  operators).  The 
more  difficult  and  germane  issues  are  the 
degrees  of  behavioral  and  logical  detail 
built  into  a  simulation.  Two  examples  help 
to  show  what  is  considered  possible  in  that 
regard  and  with  current  technology. 

The  Synthetic  Theater  of  War  ’97 
(STOW  97)  program  is  building  an  ambi¬ 
tious  and  comprehensive  knowledge  base 
of  doctrinal  behaviors.  Conceptually,  the 
synthetic  commander  will  “know”  and  be 
able  to  employ  the  doctrinally-correct  tacti¬ 
cal  response  to  changes  in  his  own  or  the 
enemy’s  order  of  battle,  to  changes  in  the 
environment,  and  to  enemy  actions. 

As  far  as  I’m  aware,  STOW  97  will  be 
the  high-water  mark  in  behavioral  detail  in 
a  simulation.  At  the  same  time,  the  draft 
Technical  Requirements  Document  for 
JSIMS  may  indicate  that  still  more  is  need¬ 
ed,  in  terms  of  both  behavioral  and  logical 
detail: 

[Section  3. 1.2. 1.3,  “Behavior”]  “...this 


behavior  [of  forces  and  their  comman¬ 
ders]  shall  include  the  ability  to  'make 
mistakes’  and  the  degradation  of  unit 
and  human  and  unit  effectiveness  due 
to  conditions  experienced  during  mili¬ 
tary  operations.  The  latter  shall  include 
factors  such  as  combat  experience, 
level  of  training,  fatigue,  social,  reli¬ 
gious,  moral  [sic],  environmental  con¬ 
ditions  (e.g.,  climate,  weather,  pres¬ 
ence  of  NBC  contaminants),  nutrition, 
sleep  deprivation,  and  political  influ¬ 
ence’"  [emphasis  added].  ^ 

There  are  two  problems  with  the  level 
of  detail  being  sought.  First,  reality 
doesn”t  exist  within  the  computer.  The 
abstraction  of  reality  that  does  exist  within 
the  computer  is  formed  by  coded  algo¬ 
rithms,  control  structures,  and  data  han¬ 
dling  and  transformation  routines. 

At  each  level  of  detail,  we  must  code 
the  cause-and-effect  relationships  between 
and  among  variables.  We  also  must  code 
any  cause-and-effect  relationships  between 
variables  at  different  levels  of  detail  or 
scale. 

For  example,  assume  we  have  been 
able  to  algorithmically  relate  the  effect  of 
sleep  deprivation  on  a  single  platoon’s 
combat  effectiveness.  We  then  need  to 
code  the  relationship  of  that  one  platoon’s 
effectiveness  on  the  effectiveness  of  its 
company.  And  so  on,  for  as  far  up  the 
organizational  scaling  ladder  as  there  is  an 
impact  (presumably,  “all  the  way,”  or  the 
individual  platoon’s  effectiveness  didn’t 
matter  in  the  first  place). 

The  problem,  obviously,  is  that  to  code 
cause-and-effect  relationships,  we  must 
first  know  them  and  be  able  to  express 
them  in  algorithmic  form. 

To  appreciate  the  full  complexity  of 
that  problem,  consider  that  none  of  the 
degradation-inducing  variables  in  the 
above  JSIMS  example  are  likely  to  be 
independent  of  the  others  in  terms  of  their 
combinatorial  effects,  and  that  virtually  all 
are  certain  to  be  nonlinear  in  their  effects  in 
the  real  world. 

We  could,  of  course,  fund  a  research 
program  specifically  to  develop  and  quan¬ 
tify  the  necessary  relationships.  It  would 
be  a  significant  program.  Or  by  default, 
the  Government  or  developer  could  simply 
invent  the  relationships. 

As  long  as  the  system  were  to  be  used 
purely  for  training  and  education,  the 

(See  OPENERS,  j!7.  26) 
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default  option  could  be  satisfactory  —  a 
panel  of  subject  matter  experts  (SMEs) 
could  be  convened  and  could  develop 
effectiveness  response  curves  that  should 
ensure  that  the  resulting  simulation  reflect¬ 
ed  “roughly  right”  sensitivity  (at  least,  one 
hopes,  with  regard  to  the  direction  of 
change  in  effectiveness)  to  the  behavioral 
variables.  The  realities  of  mutual  depen¬ 
dence  and  nonlinearity  would  almost  cer¬ 
tainly  be  ignored,  but  might  not  be  signifi¬ 
cant  in  light  of  the  fact  that  the  data  were  to 
be  invented  in  the  first  place. 

The  results  would  certainly  pass  a  face 
validation,  and  could  prove  highly  useful 
—  and  appear  exceedingly  realistic  —  for 
training.  But,  would  we  want  to  see  the 
results  used  for  the  evaluation  of  opera¬ 
tions  plans,  or  to  support  acquisition  or 
T&E  decisions? 

The  point  is  that  there  are  real  risks  — 
potentially  measured  in  lives  lost  —  to 
pushing  detail  and  causal  relationships 
below  the  level  for  which  we  have  real 
data. 

The  second  problem  is  that  if  a  greater 
degree  of  detail  is  to  have  any  meaning, 
then  the  results  of  the  simulation  must 
exhibit  some  sensitivity  to  the  more 
detailed  variables.  But  if  the  results  are 
sensitive  to  highly  detailed  variables,  then 
a  specific  result  may  be  valid  only  for  the 
unique  case  that  exhibits  the  specific  values 
of  the  variables  that  generated  that  result. 

And  —  this  is  where  the  laws  of  proba¬ 
bility  work  against  us  —  the  probability  of 
observing  that  same  specific  set  of  discrete 
conditions  in  a  multidimensional,  essential¬ 
ly  continuous  distribution  surface  is  infi¬ 
nitely  small  —  effectively  zero. 

Again,  outside  of  the  training  environ¬ 
ment  there  would  be  real  risks  in  using  the 
results  of  highly  detailed  simulations, 
unless  we  have  the  luxury  of  running  suffi¬ 
cient  replications  and  sensitivity  analyses 
to  ensure  that  the  results  are  robust. 

There  is  in  fact  another  dimension  to 
the  first  problem  above,  but  it’s  one  that 
also  points,  I  think,  to  a  solution.  At  each 
successively  greater  level  of  detail,  we  are 
faced  with  the  problem  of  having  to  devel¬ 
op  more  —  and  probably  more  complex  — 
cause-and-effect  relationships. 

But,  no  matter  how  many  such  rela¬ 
tionships  we  code  into  a  model,  we  “code 
out”  far  more. 


The  very  act  of  coding  ascribes 
absolute  certainty  to  those  relationships  in 
the  synthetic  abstraction  of  the  world  with¬ 
in  the  computer.  Even  if  the  relationships 
are  stated  as  probabilistic  algorithms,  they 
remain  certain  inside  the  computer.  No 
other  relationships  are  possible;  no  others 
are  permitted  to  exist. 

In  an  absolutely  deterministic  real 
world,  one  in  which  all  cause-and-effect 
relationships  were  known  with  certainty, 
there  would  be  no  theoretic  problem.  In  a 
highly  dynamical,  nonlinear,  even  turbu¬ 
lent  system  (e.g.,  warfare),  the  problem  is 
massive.  (To  anyone  who  has  doubts  that 
warfare  is  all  three  of  those,  even  at  the 
theater  level,  I  strongly  recommend  Admi¬ 
ral  Sandy  Woodward’s  memoirs  of  the 
Falklands  War,  “One  Hundred  Days.”) 

As  a  result,  (here  I’ll  go  well  out  on  a 
theoretic  limb,  and  see  who  cuts  it  off 
behind  me)  I  believe  there  exists  a  bound¬ 
ary  condition  on  detail  (or  “granularity”)  in 
simulations  such  that  adding  detail  beyond 
the  boundary  is  actually  counterproductive 
to  predictive  accuracy. 

I  also  think  it  means  that  “scaling  up” 
entity-level  models  to  answer  campaign- 
level  and  beyond  analytic  questions  is  an 
approach  doomed  to  failure,  simply 
because  much  if  not  most  of  the  turbulence 
in  warfare  exists  in  the  interactions  among 
entities  and  between  levels,  and  is  unpre¬ 
dictable  inform,  direction,  and  timing. 

As  heuristic  evidence,  consider  that  one 
of  the  principal  characteristics  of  a  chaotic 
system  is  sensitive  dependence  on  initial 
conditions.  “Initial  conditions”  is  itself  a 
fluid  concept;  i.e.,  it  can  be  taken  at  any 
point  in  an  event  stream,  not  necessarily  at 
the  first  point  —  if  that  is  even  known.  In 
Chaos  Theory,  Lorenz  recognized  that  it 
was  minute  variations  in  initial  conditions 
in  a  set  of  deterministic  equations  that 
caused  wildly  chaotic  behavior  in  a  weath¬ 
er  simulation. 

In  warfare,  the  examples  may  not  be  so 
neatly  stated,  but  they  are  legion.  As 
examples  of  the  sensitive  dependence  on 
initial  conditions  in  warfare,  consider: 

•  how  would  the  course  of  war  in  the 
Pacific  have  changed  if  the  Pacific 
Fleet  carriers  had  been  caught 
along  with  the  battleships  at  Pearl 
Harbor? 

•  how  might  the  course  of  war  in  the 
Pacific  have  changed  if  our  planes 
had  not  found  the  Japanese  carriers 


at  Midway  until  as  little  as  an  hour 
later  (i.e.,  with  planes  launched, 
instead  of  on  deck,  fueled,  and 
changing  out  weapons  loads)? 

•  how  might  the  course  (and  possibly 
the  outcome)  of  war  have  changed 
in  the  Falklands  if  the  Argentinean 
pool  of  air-launched  Exocet  mis¬ 
siles  —  tremendously  effective  at 
the  engagement  level  even  though 
mismanaged  at  the  campaign  level 
—  been  much  greater  than  five? 

Shifting  gears,  can  we  take  anything 
positive  out  of  all  the  above?  I  think  so. 
While  it  strongly  argues  against  the  utility 
of  scaling  up  entity-level  models  for  the¬ 
ater  level  and  beyond  analytic  applications, 
I  think  it  also  argues  for  entity-level  model¬ 
ing  (stochastic  entity-level  modeling)  in 
general  to  address  engagement-level  ques¬ 
tions. 

It  principally  argues  against  detail  for 
its  own  sake  in  our  models.  Some  general 
rules  of  thumb  may  be  useful  in: 

(1)  If  there  are  no  hard  data  to  sup¬ 
port  specific  cause-and-effect 
relationships  at  some  level  of 
detail,  either  move  to  a  higher 
level  or  don’t  apply  the  model  in 
any  applications  requiring  predic¬ 
tive  accuracy 

(2)  Only  specifically  include  vari¬ 
ables  that  you  can  in  the  real 
world  plan  for,  observe,  or  con¬ 
trol  (as  an  example,  if  my  simula¬ 
tion  is  to  support  weapons  sys¬ 
tems  acquisition  decisions,  and  I 
can’t  plan  for  or  control  the  reli¬ 
gion  or  politics  of  my  adversaries, 
those  particular  variables  would 
be  counterproductive  in  my  simu¬ 
lation.  Conversely,  if  my  simula¬ 
tion  is  being  used  to  train  Special 
Operations  Forces,  and  there  are 
unique  and  tactically  useful 
aspects  of  the  religions  of  my 
potential  adversaries,  and  if  I 
know  those  aspects  and  their 
effects,  then  I  could  be  justified  in 
including  them. 

Aggregate-Level,  Again 

None  of  the  above  means,  though,  that 
we  must  fall  back  on  aggregate-level  simu¬ 
lations  as  the  best  way  to  go  for  campaign- 
level  and  above  applications.  I’ve  helped 
to  build  a  couple,  used  several,  and 
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reviewed  a  few.  Fve  yet  to  come  across 
even  one  that  I  could  say  was  predictively 
valid  or  accurate  in  any  absolute  sense. 
That  means  that  “accuracy”  was  absolutely 
dependent  on  the  skills,  knowledge,  and 
objectivity  of  the  analysts. 

The  track  record  has  not  been  encour¬ 
aging  in  that  regard.  Along  with  all  the 
problems  mentioned  earlier  with  respect  to 
the  higher  levels  of  warfare,  must  go  the 
potential,  particularly  prevalent  in  aggre¬ 
gate-level  simulations,  for  subjective  inter¬ 
ference  precluding  predictive  accuracy. 

One  example  should  suffice.  A  couple 
of  years  ago,  in  a  theater-level  assessment, 
I  observed  a  surprisingly  high  level  of  attri¬ 
tion  of  enemy  air  assets  in  an  extremely 
short  time  interval,  against  essentially  no 
friendly  air-to-air  losses. 


On  closer  examination,  it  turned  out 
that  the  friendlies  were  engaging  the 
enemy  aircraft  at  maximum  weapons 
range,  given  detection,  and  that  there  was  a 
Blue  advantage  in  detection  range  capabili¬ 
ty.  Rules  of  Engagement?  What  Rules  of 
Engagement?  ROEs  were  not  modeled 
explicitly,  and  so  were  ignored  completely. 

The  correct  answer,  and  one  easily 
implemented  in  aggregate-level  simula¬ 
tions,  would  have  been  to  “turn  down”  the 
range  capability  of  the  missiles  to  reflect 
realistic  ROE  restrictions. 

Wrong!  As  I  was  told,  the  “correct” 
answer  would  have  so  degraded  the  effec¬ 
tiveness  of  the  friendly  missiles  that  the 
entire  value  —  and  thus  programmatic 
future  —  of  a  set  of  AAW  capabilities 
could  have  been  threatened. 

Is  it  any  wonder  that  we  have  trouble 
convincing  anyone  of  the  operational  value 
of  some  capabilities?  Much  of  their 
value,  of  course,  is  due  to  their  synergistic 
effect  with  our  weapons  systems.  That 
value  can  only  be  fully  demonstrated  (in 
attrition  models)  by  accurately  showing 
how  much  more  poorly  the  weapons  sys¬ 
tems  perform  in  the  absence  of  particular 
capabilities.  As  stated  earlier,  the 
potential  for  subjective  “tuning”  is  particu¬ 
larly  high  in  aggregate-level  simulations. 


Solution? 

I  think  we  need  to  take  a  much  different 
approach  to  campaign-level  and  above 
modeling,  again  if  predictive  accuracy  is 
the  goal.  And  the  result  may  not  look  at  all 
“realistic.” 

Imagine  that  I  want  to  build  a  model  of 


the  U.  S.  economy,  one  that  would  tell  me 
whether  I  should  invest  in  specific  stocks, 
particular  types  of  mutual  funds,  bonds,  or 
commodities.  I  could,  given  enough 
research  funds,  build  a  detailed  entity-level 
(consumer-level)  simulation  of  the  econo¬ 
my  (and  maybe  the  rest  of  the  world’s 
economies  too,  because  of  the  interdepen¬ 
dencies).  Entity-level  population  models 
could  be  integrated  into  the  whole. 

I  could  then  watch  resources  migrate 
into  certain  economic  sectors  relative  to 
others  as  time  went  on.  It  could  certainly 
look  realistic.  I  could  watch  Dick  queue  up 
at  the  grocery  store,  watch  Jane  decide  on  a 
Ford  rather  than  a  Chevy  based  on  some 
market  statistic  (next  iteration  she  might 
take  the  Chevy,  or  a  Harley). 

No  matter  how  detailed  I  made  it,  does 
anyone  believe  it  would  be  predictively 
accuratel  How  about  if  we  tried  it  at  the 
aggregate  level?  Blocks  of  consumers 
could  interact  with  blocks  of  goods  and 
services... 

Fortunately,  in  economic  modeling, 
we've  got  data.  Reams  of  data.  Which 
means  we  can  use  the  tools  of  statistics  and 
the  computer  to  make  sense  out  of  the  data 
—  to  let  the  data  dictate  the  model 

No,  econometrics  is  far  from  the  ideal 
answer,  even  in  applications  where  the  data 
are  sufficient.  Econometric  models  carry 
heavy  baggage,  not  the  least  of  which  are 
rigid  assumptions  about  data  and  error  dis¬ 
tributions.  But  I’m  convinced  the  basic 
concept  is  right  for  campaign-level  appli¬ 
cations  and  above;  we've  got  to  let  the  data 
drive  and  shape  the  models. 

We  also  don’t  know  yet  how  to  do  that 
in  a  relatively  data-sparse  environment 
such  as  warfare.  We  need  research  into 
that  particular  field  (Econometric  Model¬ 
ing  in  a  Data-Sparse  Environment?). 

Much  of  the  discussion  here  also  has 
pointed  up  the  apparently  strong  analogies 
between  warfare  and  turbulence,  warfare 
and  “chaos.”  We  need  serious,  in  depth 
research  into  Chaos  Theory  applied  to  war¬ 
fare  modeling.  I  am  personally  convinced 
that  real-world  warfare  results  will  be 
found  to  be  much  better  explained  and 
understood  —  even  modeled  —  by  the 
nonlinear  mathematics  of  Lorenz,  Huber- 
man,  Mandell  et  al,  rather  than  by  the 
determinism  characterized  by  Lanchester. 
Please  let  us  know  your  thoughts. 


Robert  Eberth 
13371  Meadowsweet  Drive 
Fairfax,  VA  22033-1108 
eberthr@  smtp-gw.spawar.navy.mil 

Footnotes 

'  As  an  aside,  when  I  first  saw  this  list  of 
“soft  factors”  to  be  included  in  the  simu¬ 
lation,  I  was  struck  by  the  absence  of  the 
soft  factor  that  is  arguably  the  most 
important  of  all.  I’m  indebted  to  the  late 
Colonel  Trevor  DuPuy,  USA,  who  stat¬ 
ed  at  a  MORS  More  Operational  Realism 
in  the  Modeling  of  Combat  (MORI- 
MOC)  working  group  several  years  ago, 
“the  most  pervasive  element  influencing 
performance  in  combat  is  fear  in  a  lethal 
environment."  If  you’ve  been  there,  you 
know  he’s  right.  It’s  also  the  most  radi¬ 
cally  nonlinear  element  I  can  imagine.  O 


NUMBERS  FROM  COMBAT 

(continued from  p.  7) 

necessary  detail.  We  don’t  know  whether 
all  these  objects  have  important  effects,  so 
we  have  a  choice.  We  can  aggregate  their 
behavior  via  another  object  to  avoid  getting 
our  combinatorial  lunch  eaten,  in  which 
case  we  would  have  been  better  off  using 
the  body  of  aggregate  methods  we  know 
and  only  decomposing  as  necessary.  Or, 
we  can  spend  a  great  deal  of  time  and 
money  gathering  data  on  these  objects  only 
to  discover  we  either  can’t  model  the  inter¬ 
actions  or  we’ve  mn  out  of  time  and  money 
-  in  which  case  we’ve  accomplished  noth¬ 
ing.  Tell  me  again  why  we’re  doing  this? 

I  may  be  hopelessly  old-fashioned,  but 
this  is  what  I  think  Operations  Analysts  are 
supposed  to  do:  we  should  build  abstrac¬ 
tions  of  reality,  check  that  they  make  sense 
for  attainable  limiting  cases,  refine  them  for 
regions  of  interest,  and  use  them  to  support 
decision  making.  In  Marshall  and  Garrett’s 
world,  I  pick  a  method,  decompose  my 
model  to  the  lowest  possible  level,  gather 
an  extraordinary  amount  of  data,  and  hope 
that  the  secondary  issues  of  whether  my 
model  is  correct  will  sort  themselves  out.  I 
believe  that  object-oriented  approaches 
have  their  place,  and  I  suspect  I  will 
employ  these  methods  sometime  during  my 
career.  But  objects  are  just  another  ham¬ 
mer,  and  I  don’t  use  hammers  on  things 

(See  NUMBERS  FROM  COMBAT  p.  28) 
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NUMBERS  FROM  COMBAT  into  the  inherent  capacity  of  various  OR  attempted):  on  “objects,”  “relationships,” 

(continued from  p.  27)  approaches  for  fidelity  to  the  real  world  “variables”  or  “things”  and  their  causal 

they  purport  to  represent.  The  debate  over  interactions,  or  whatever.  Such  judgments 

that  don’t  look  like  nails.  “object-orientation”  (which  is  every  bit  as  usually  either  originate  essentially  with  us 

fundamental  as  Yost  suggests)  seems  to  or  deri ve  from  a  given  OR  approach’ s  theo- 

Editor^s  Note:  I  welcome  Major  Yost  as  lead  to  another  inquiry:  into  our  judgments  retical  structure.  Inquiry  into  the  character 

the  first  independent  contributor  to  this  col-  as  to  What  out  there  in  the  real  world  in  fact  and  empirical  supportability  of  the  phe- 

umn.  His  article  forcefully  raises  both  constitute  the  ''phenomena”  and  their  char-  nomena  as  judged  (whether  originally  or 

practical  and  theoretical  concerns  about  a  acteristics  to  be  represented.  Such  judg-  derivatively)  is  another  of  the  column’s 

growing  body  of  effort  in  the  OR  commu-  ments  determine  whether  and  how  we  purposes.  Reader  comment  is  invited  on 

nity.  As  such,  his  presentation  fits  nicely  focus  our  analytic  attention  from  the  first  either  topic  of  inquiry,  of  course  including 

into  one  of  the  column’s  purposes:  inquiry  instance  (i.e.,  before  any  representations  are  Kirk’s  argument.  O 
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MORS  PRESroENT 

(continued from  p.  3) 

“Foundations”  theme  discussed  above  and 
the  slate  of  Special  Meetings  for  the  next 
year  that  leads  us  into  the  Quantico  Sympo¬ 
sium. 

Our  Special  Meetings  Committee  will 
continue  under  Dr  Stuart  Starr’s  leadership 
this  year  and  has  an  ambitious  slate  of  pro¬ 
posed  meeting  topics  and  work  underway 
for  meetings  right  around  the  comer  —  this 
fall  and  winter.  Please  feel  that  you  are  an 
integral  part  of  both  the  65^  MORSS  and 
the  Special  Meetings  this  year  and  GET 
INVOLVED!!  The  MORS  Officers  and 
Board  desire  to  focus  “outside”  on  the  con¬ 
duct  of  quality  meetings  and  publications 
this  year.  We  are  moving  quickly  under 
Jackie  Henningsen’s  Joint  Analysis  SAG 
to  get  a  “Quick  Response  Analysis”  meeting 
in  place  for  the  week  of  1  October  1996,  a 
most  ambitious  plan!!  More  detail  on  the 
complete  list  of  Special  Meetings  in  plan¬ 
ning  for  this  year  are  contained  in  the 
VP(MO)  article  “VEEPS  PEEP”  on  page  5. 

The  Sponsor’s  and  Past  Presidents  Lun¬ 
cheon  Meetings  conducted  during  the 
MORSS  provided  us  with  some  valuable 
insights  into  what  is  HOT  for  the  next  year. 
One  suggestion  is  to  start  working  on  an 
International  Symposium  for  participation 


by  selected  allies  to  be  conducted  perhaps 
every  other  year.  That  meeting  could  grow 
to  be  of  the  size  and  quality  of  our  annual 
Symposium.  Across  the  board  we  heard  a 
consistent  concern  with  meaningful  analysis 
to  support  the  new  joint,  coalition  and  inter¬ 
operability  environments  —  as  well  as 
Operations  Other  than  War.  There  is  con¬ 
cern  that  we  can  provide  the  right  tools  for 
collaborative  analysis  and  decision  making 
under  uncertainty.  Watch  more  for  these 
topics  as  we  unfold  our  special  meetings 
agenda  and  prepare  for  Quantico  in  ’97. 

The  following  table  lists  the  new  officers 
and  their  phone  and  email  addresses.  Don’t 
hesitate  to  call  me  directly  or  any  other 
member  of  our  Board  of  Directors  and  man¬ 
agement  team.  We  welcome  your  input  and 
active  involvement  in  this,  your  Military 
Operations  Research  Society.  The  new 
MORS  Organization  is  listed  in  the  follow¬ 
ing  diagram.  Get  to  know  your  VP’s  and 
Committee  Chairs  and  let  us  hear  from  you. 

I  consider  it  to  be  one  of  the  most  impor¬ 
tant  events  in  my  professional  life  to  have 
been  elected  to  serve  for  the  next  year  as 
President  of  your  Society.  It  is  an  honor  and 
privilege  to  join  the  ranks  of  the  illustrious 
former  office  holders  of  this  position.  I’m 
looking  forward  to  a  great  MORS  year 
together!  O 
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W&A 

(continued  from  p.  1) 

starting  point  and  catalyst  for  a  wide- 
ranging  discussion  of  V&V  issues  and 
activities  by  the  entire  Colloquium.  The 
remainder  of  this  article  highlights  the 
discussions  involving  several  common 
threads  that  ran  throughout  the  confer¬ 
ence.  In  general: 

•  Both  the  DoD  and  non-DoD  M&S 
communities  share  similar  objectives, 
e.g.,  credible  M&S; 

•  Both  the  DoD  and  non-DoD  M&S 
communities  are  experiencing  similar 
problems,  e.g.,  data; 

•  The  tools  and  methodologies  used  by 
both  communities  (excluding  DIS  and 
ADS)  are  fairly  equivalent. 

Data  and  Test  Sufficiency 

Almost  universally,  presenters  indicat¬ 
ed  that  their  modeling  or  V&V  efforts 
were  hampered  by  two  interrelated  prob¬ 
lems:  the  lack  of  both  sufficient  (real- 
world  systems)  testing  and  data.  In 
recent  years  there  has  been  a  reduction  in 
testing  due  to  many  factors  such  as 
reduced  budgets,  increased  test  costs, 
safety  considerations,  range  constraints, 
etc.  This  has  led  to  a  corresponding  loss 
of  data  collection  opportunities  afforded 
by  the  test  process.  (Even  when  testing  is 
available,  there  are  two  instances — fideli¬ 
ty  mismatches  between  model  and  test 
data,  and  corruption  of  data  via  the  data 
collection  process — where  the  test  data  is 
later  found  to  be  unusable  for  V&V  pur¬ 
poses.  It  is  not  always  possible  to  deter¬ 
mine  a  priori  if  certain  test  data  will  fall 
into  one  of  these  categories.) 

Besides  reductions  in  testing,  other 
reasons  for  this  dearth  of  quality  data — 
similar  to  that  experienced  within  DoD — 
include: 

•  The  data  for  a  particular  problem 
domain  is  too  difficult  to  collect  or 
does  not  exist.  This  was  illustrated  in 
Dr.  Kleinberger’s  presentation  of 
Human  Anatomic  Finite  Element  Mod¬ 
els,  when  he  indicated  that,  in  certain 
test  scenarios,  it  was  infeasible  to 
obtain  and  use  live  human  or  animal 
subjects,  and  that  surrogate  subjects 
(crash  dummies  or  human  cadavers) 
were  incapable  of  providing  realistic 
data. 


•  The  data  is  available,  but  the  collection 
method  corrupts  the  data  being  collect¬ 
ed.  An  example — a  sensor  inserted 
into  a  flow  field  would  itself  disrupt 
the  flow  field  it’s  trying  to  characterize. 
This  disruption  may  alter  the  test  data 
enough  to  make  it  unusable  for  validat¬ 
ing  certain  high-fidelity  model  applica¬ 
tions. 

•  The  data  is  available  but  is  inappropri¬ 
ate  for  use  in  model  validation  efforts. 
Mr.  Hicks  noted,  in  his  presentation  of 
dispersion  models,  that  actual  low-alti¬ 
tude  atmospheric  wind  patterns  are 
both  extremely  random  and  localized. 
This  localized  data  is  easiest  to  collect 
even  though  wide  area  wind  patterns 
are  generally  of  prime  interest.  The 
issue:  is  it  possible,  or  appropriate,  to 
validate  a  model  when  there  is  a  mis¬ 
match  between  the  data  and  model 
fidelities  (in  this  case,  using  detailed 
localized  wind  data  to  validate  a  more 
aggregated,  wide  area  wind  model.) 

•  The  data  is  available  but  is  considered 
too  sensitive  for  release  outside  of  the 
owning  organization  due  to  public  pol¬ 
icy,  legal,  corporate  proprietary,  or 
similar  considerations.  These  situa¬ 
tions  were  observed  in  Dr.  KnepelTs 
presentation  on  modeling  AIDS  trans¬ 
missivity  (public  policy)  and  Mr. 
Neat’s  rivet  crack  propagation  presen¬ 
tation  (where  the  aerospace  industry 
was  reluctant  to  share  its  proprietary 
models  or  data  because  of  possible  use 
against  them  in  future  litigation). 

•  Data  sets  provided  from  multiple 
authoritative  and  pseudo  authoritative 
sources  contain  data  voids,  conflicts, 
or  inconsistencies.  The  opposite  situa¬ 
tion  also  applies:  data  production 
sources  may  refuse  release  of  these 
problem  datasets — even  with  appropri¬ 
ate  caveats — due  to  potential  data  mis¬ 
use  or  damage  to  the  organization’s 
reputation  once  the  data  are  beyond 
their  control. 

Face  Validation 

One  result  of  this  lack  of  data  is  the 
heavy  reliance  on  expert  face  validation 
and  peer  review  as  means  of  establishing 
model  credibility.  Colloquium  partici¬ 
pants  felt  these  techniques  could  be  made 
more  effective  if  they  were  repeatedly 
accomplished  over  time.  Reviewers 
bring  to  the  model  evaluation  task  both 


their  subject  matter  expertise  and  some 
level  of  familiarity  with  the  strengths  and 
weaknesses  of  the  best  models  currently 
used  within  their  problem  domain. 
Repeated  successful  face  validation  of  a 
model  for  different  aspects  of  the  prob¬ 
lem  domain  by  such  review  panels  would 
raise  the  model’s  credibility  within  that 
community. 

While  widely  accepted,  Colloquium 
participants  noted  that  these  approaches 
have  their  own  limitations: 

•  They  assume  you  can  make  an  ade¬ 
quate  observation  of  the  phenomenon 
being  modeled.  However,  the  difficulty 
in  observation  is,  at  times,  the  cited 
justification  for  using  M&S  (to  provide 
insights  into  the  phenomenon  being 
observed  or  modeled.) 

•  They  assume  that  there  are  accepted 
community  standards  against  which 
these  models  can  be  judged.  However, 
it  was  noted  that  there  is  intense  debate 
within  many  disciplines  as  to  the  theo¬ 
retical  constructs  that  correctly  portray 
the  behavior  of  their  problem  domain. 
Such  debate  would  also  extend  to  the 
review  of  models  based  on  these  con¬ 
structs. 

One  method  to  possibly  offset  the 
observability  problem  is  through  the  use 
of  appropriate  visualization  tools.  These 
tools  would  permit  the  observer  to  see 
what  is  happening  with  the  model  over 
time.  This  would  benefit  the  analysts  that 
are  using  these  models  to  gain  better 
understanding  of  very  fine  nuances  with¬ 
in  their  problem  domains.  (Later,  these 
tools  would  be  particularly  effective  in 
building  the  accreditation  authority’s 
confidence  in  the  model’s  credibility.) 
One  note  of  caution:  the  analyst  must 
know  where  or  what  in  the  model  to  visu¬ 
alize.  The  tool  can  provide  misleading 
information  if  the  visual  is  set  incorrectly. 

It  was  also  suggested  that  practitioners 
not  automatically  limit  themselves  to  face 
validation  if  they  are  not  able  to  compare 
model  output  with  sufficient  instrument¬ 
ed  live  test  data.  For  example,  one 
method  to  increase  model  confidence  is 
by  stimulating  the  model  in  such  a  man¬ 
ner  that  its  outputs  can  be  analytically 
solved  and  then  compare  calculated  ver¬ 
sus  model  outputs.  Another  means  of 
establishing  a  model’s  goodness  could  be 
through  use  of  Dr.  Knepell’s  experimen- 
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tal  design  methodology.  (It  was  noted 
that  more  work  was  needed  to  develop 
better  measures  of  goodness,  as  statistical 
point  estimates  such  as  bias,  variance, 
and  correlation  are  not  adequate  for  this 
task.  It  was  suggested  that  fuzzy  logic 
techniques  be  used  as  a  means  of  deter¬ 
mining  model  goodness  metrics.)  Often 
overlooked,  the  sensitivity  analysis  is  one 
of  the  most  useful  tools  of  the  analyst  try¬ 
ing  to  validate  a  model.  The  bottom  line 
is  that  there  are  other  available  tech¬ 
niques,  and  practitioners  should  make 
good  use  of  them. 

V&V  of  Model  Hierarchies 

One  of  the  lessons  learned  presented 
during  Mr.  Gordon’s  discussion  of  rail 
car  wheel  heat  treatment  simulations  was 
that  no  one  model  does  it  all.  This  illus¬ 
trates  a  trend  within  many  M&S  commu¬ 
nities  over  the  past  several  years  towards 
the  use  of  model  suites  or  toolkits  to 
accomplish  problem  domain  analysis. 
The  composition  of  these  suites  support 
problem  analysis  vertically  (using  models 
that  address  the  problem  domain  in 
increasing  levels  of  fidelity),  horizontally 
(using  models  of  similar  fidelity  that 
address  different  aspects  of  the  problem 
domain),  or  cross-cutting  (that  is,  for  an 
analysis  spanning  several  problem 
domains.) 

Within  these  suites,  outputs  from  one 
model  are  often  used  as  input  to  another 
model(s).  Each  model  will  produce  data 
at  a  certain  level  of  error.  Data  transfor¬ 
mations  and  interfaces  facilitating  data 
flow  between  the  models  will  increase 
this  error.  This  raises  a  question:  how 
should  practitioners  conduct  V&V  of 
integrated  modeling  suites,  and  how 
would  it  differ  from  V&V  of  stand  alone 
models?  The  Colloquium  participants 
concluded  that  V&V  should  be  conduct¬ 
ed  in  two  steps:  first,  conduct  V&V  on 
individual  models  within  the  suite,  then 
conduct  V&V  on  the  model  suite  in  the 
configuration  used  during  actual  analysis 
(treat  it  as  a  black  box  model.)  This  is  an 
approach  similar  to  that  recommended 
for  the  V&V  of  DIS  environments. 

Colloquium  members  also  discussed 
the  linkage  (if  any)  between  V&V  activi¬ 
ties  and  accreditation  (that  is,  meeting  the 
decision-maker’s  needs).  Many  Collo¬ 
quium  members  appeared  to  support  the 
position  that  the  needs  of  the  decision 


makers  must  be  the  primary  guide  when 
planning  V&V  activities.  This  would 
drive  a  need  to  interactively  develop 
requirements  and  acceptance  criteria  with 
the  decision  maker,  then  validate  to  that 
criteria 

Legacy  Models 
and  the  V&V  Process 

Legacy  models,  as  defined  by  DoD, 
are  models  developed  in  the  past  which 
are  still  in  use  that  were  not  implemented 
using  today’s  standards.  (Some  legacy 
models  have  been  modified  to  meet  the 
current  standards,  thereby  extending  their 
usefulness  and  interoperability  with 
newer,  standards  based  models.)  Even 
though  these  models  that  have  been  used 
successfully  for  many  years  in  a  given 
problem  domain,  they  must  nonetheless 
be  subjected  to  the  VV&A  process  when 
they  are  used  for  new  purposes.  This  was 
one  of  the  lessons  learned  highlighted 
during  Mr.  Keltch’s  Tertiary  Oil  Recov¬ 
ery  Information  System  (TORIS)  presen¬ 
tation.  The  TORIS  modeling  suite  had 
been  used  successfully  in  analyzing 
petroleum-related  problems  at  the  nation¬ 
al  level.  In  a  clear  departure  from  previ¬ 
ous  usage,  TORIS  was  used  in  a  royalty 
reduction  study  aimed  at  increasing  oil 
production  (and  thus  increasing  the  in- 
ground  reserves’  royalty  producing  capa¬ 
bility)  within  Wyoming.  Initial  model 
results  were  at  variance  with  the  antici¬ 
pated  output.  Subsequent  model  evalua¬ 
tion  (which  greatly  benefited  from  the 
use  of  proprietary  petroleum  industry 
data)  resulted  in  several  changes  that 
addressed  the  resolution  problems  and 
corrected  the  model  output.  Policy  deci¬ 
sions  made  as  a  result  of  the  TORIS-sup- 
ported  study  substantially  raised  the  value 
of  these  in-ground  reserves. 

Cost  of  V&V  Activities 

Model  V&V  and  IV&V  processes  are 
expensive  propositions.  The  Colloquium 
participants  focused  on  the  issue  of  what 
should  be  spent  on  V&V.  It  was  the  gen¬ 
eral  consensus  of  the  participants  that 
V&V  costs  are  situation-dependent, 
based  on  such  factors  as  study  objectives, 
model  complexity  and  the  V&V  level  of 
effort  (LOE)  needed  to  reach  a  predeter¬ 
mined  confidence  level.  Two  LOE  meth¬ 
ods  were  mentioned: 


•  The  Criticality  Analysis  and  Risk 
Assessment  (CARA)  method  used  by 
NASA  that  rates  each  software  func¬ 
tion  against  seven  weighted  risk  and 
criticality  drivers — is  one  LOE  (but  not 
cost)  estimation  tool.  (CARA  is  a  soft¬ 
ware  IV&V  tool — it  makes  no  provi¬ 
sion  to  the  V&V  of  the  underlying  con¬ 
ceptual  model,  so  it  would  probably 
underestimate  LOE  required  for  full 
model  V&V.) 

•  Colloquium  participants  that  are  active 
in  the  distributed  interactive  simulation 
(DIS)  VV&A  arena  noted  that  a  DIS 
costing  tool  capable  of  providing  cost 
estimates — as  a  percentage  of  DIS 
development  costs — for  a  user-speci¬ 
fied  DIS  V&V  LOE  is  currently  under 
development.  An  extension  of  this 
work — to  develop  a  generic  costing 
process — is  also  part  of  the  DMSO 
VV&A  initiative. 

Budgeting  for  V&V  presents  an  inter¬ 
esting  program  management  problem.  In 
order  to  provide  V&V  funding  during 
program  start-up,  you  must  somehow 
accurately  link  (estimated)  V&V  and 
overall  software  costs.  One  of  our  Collo¬ 
quium  presenters  suggested  that,  because 
of  the  variable  nature  of  these  require¬ 
ments,  V&V  should  not  be  estimated  as  a 
fixed  percentage  of  the  software  develop¬ 
ment  costs.  Program  managers  would  be 
hesitant  to  agree  to  a  variable  funding 
scheme  for  model  V&V.  However,  if 
over  time  you  could  demonstrate  the 
effectiveness  of  V&V  to  these  managers, 
through  the  use  of  well-documented  suc¬ 
cess  stories,  they  might  be  more  inclined 
to  approve  front-loading  a  V&V  funding 
wedge.  As  an  example:  Mr.  Gordon’s 
presentation  on  shakedown  stress  estima¬ 
tion  studies  of  rail  car  wheels  showed 
that,  by  modifying  the  parameters  used  to 
determine  when  to  turn  down  these 
wheels,  one  could  save  $875,000  annual¬ 
ly  at  a  one  time  combined  cost  (model 
development,  V&V,  and  study  imple¬ 
mentation)  of  $700,000.  While  acknowl¬ 
edging  this  argument’s  effectiveness,  it 
has  thus  far  proved  difficult  to  locate  very 
many  of  these  success  stories. 

V&V  —  Personnel  Dimensions 

Any  discussion  of  modeling  activities 
will  eventually  touch  upon  how  the 
(SeeyW&kp,32} 
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(continued  from  p.  31) 

human  being  impacts  the  process  (or,  as 
stated  in  the  September  1995  edition  of 
the  PHALANX,  It’s  the  ANALYST,  Stu- 
pid!).  From  personal  experience,  Collo¬ 
quium  participants  have  found  that  in 
conferences  where  this  subject  is  dis¬ 
cussed,  the  analyst  is  always  rated  as  far 
more  important  than  modeling  tools  and 
data  to  the  successful  completion  of  a 
study.  (It  was  previously  mentioned  that 
successful  validation  using  peer  review  or 
face  validation  was  heavily  dependent  on 
the  quality  of  the  participants.)  Two 
questions  were  raised  at  the  Colloqui¬ 
um’s  wrap-up  session: 

•  What  type  of  people,  in  terms  of  skill 
sets,  should  be  considered  when 
putting  together  a  V&V  effort? 

•  How  do  you  track  and  tap  into  this 
expert  base?  Or,  more  importantly, 
how  do  you  identify  those  organiza¬ 
tions  with  the  appropriate  expertise  that 
can  provide  a  user  with  credible  model 
V&V? 

There  was  no  consensus  answer  per  se 
identified  for  either  of  these  questions. 
The  discussion  on  model  validation  (as 
used  by  DoD)  pointed  towards  a  team 
effort  that  includes  expertise  in  statistical 
methodology  and  analysis,  software 
development,  and  in  the  functional  area 
being  modeled.  The  latter  point — the 
criticality  of  competent  subject  matter 
expertise  (SME) — was  part  of  the  lessons 
learned  drawn  from  the  TORIS  and  rail 
car  heat  treatment  case  studies.  It  should 
also  be  noted  that  Colloquium  presenters, 
by  and  large,  were  highly  experienced 
subject  matter  experts  within  their  prob¬ 
lem  domains.  The  (software)  verifica¬ 
tion  step — the  subject  of  two  Colloquium 
presentations — should  be  relatively  easier 
and  more  straight  forward,  than  the  vali¬ 
dation  step.  This  is  primarily  due  to  the 
availability  of  better  verification  tools 
(such  as  Computer  Aided  Software  Engi¬ 
neering  tools.  Configuration  Management 
tools,  and  critical  path  analysis)  and  tech¬ 
niques  (Dr.  Miller’s  presentation  on  soft¬ 
ware  indicated  they  uncovered  153  spe¬ 
cific  technique  classes  during  their 
literature  review). 

The  second  question  is  of  particular 
interest  to  the  DoD  participants  (especial¬ 


ly  M&S  users).  There  are  no  known 
plans  to  establish  dedicated  V&V 
provider  organizations  within  DoD,  and 
many  DoD  organizations  do  not  possess 
an  in-house  V&V  capability.  Currently, 
the  ability  to  locate  and  assemble  the 
appropriate  V&V  team  is  a  function  of  a 
program’s  available  resources  and  knowl¬ 
edge  its  personnel  possess  on  both  the 
appropriate  functional  and  modeling 
communities.  (Editor’s  note:  since  the 
Colloquium,  there  has  been  work  within 
DMSO  to  provide  accreditation  authori¬ 
ties  with  access  to  M&S  technical  sup¬ 
port  services.) 

Standardized  Terminology 

Early  on  in  the  proceedings,  it  became 
clear  that,  while  Colloquium  members 
shared  the  same  understanding  of  the 
activities  that  constitute  model  V&V, 
they  did  not  share  a  common  terminolo¬ 
gy.  This  required  the  Colloquium  mem¬ 
bers  to  cross-map  their  definitions  with 
those  of  the  DoD  standard  definitions  of 
verification,  validation,  and  accreditation. 
It  was  the  consensus  among  Colloquium 
participants,  at  the  conclusion  of  the  con¬ 
ference,  that  words  must  have  precise 
meaning  within  the  modeling  community 
to  avoid  potential  confusion  among  its 
members. 

Parting  Comments 

The  most  perfect  V&V  methodology, 
flawlessly  executed,  is  wasted  effort 
unless  it  ultimately  increases  the  deci¬ 
sionmaker’s  confidence  in  the  model — 
and  contributes  to  their  willingness  to 
implement  the  model-supported  end 
product.  (This  is  not  meant  to  downplay 
the  critical  impact  that  the  analyst  wields 
regarding  the  decision  maker’s  accep¬ 
tance  of  the  end  product.)  This  was  the 
case  with  our  presenters: 

•  Public  policy  changes  in  several  Third 
World  countries  resulted  from  Dr.  Kne- 
pell’s  study  on  AIDS  transmissivity; 

•  The  frequency  of  aircraft  inspections 
were  influenced  by  Mr.  Neat’s  crack 
propagation  analysis; 

•  The  maintenance  of  rail  car  wheels  and 
railroad  track  was  influenced  by  the 
stress  analyses  described  by  Mr.  Gor¬ 
don; 

•  The  value  of  Wyoming’s  in-ground  oil 
reserves  royalty  producing  potential 


was  increased  because  of  Mr.  Keltch’s 
TORIS  model  analysis. 

As  one  Colloquium  participant  noted 
during  final  wrap-up  discussions,  “I 
didn’t  come  here  expecting  to  hear  the 
Golden  Nugget,  and  I  didn’t  hear  it — but 
I  got  a  lot  of  benefit  from  the  Colloqui¬ 
um.”  One  of  the  benefits  was  a  confirma¬ 
tion,  based  on  the  cases  cited  by  our  pre¬ 
senters,  that  V&V  can  contribute  to  the 
organization’s  bottom  line  mission.  In 
the  final  analysis,  that’s  what  it’s  all 
about. 
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Research  on  the  Causes  of  Dynam¬ 
ical  Instability  in  Combat  Models 
by  Julian  L  Palmore  is  available  as 
USACERL  Technical  Report  96/95. 

Abstract 

Combat  models  are  nonlinear  deter¬ 
ministic  models  of  decisionmaking 
processes  that  deal  with  attrition  of 
opposing  forces.  They  contain  dynami¬ 
cal  instabilities  that  destroy  the  robust¬ 
ness  of  a  simulation  and  interfere  with  a 
simulation’s  consistency,  fidelity,  porta¬ 
bility,  reliability,  and  validity.  Nonlinear 
behavior  in  a  model’s  design  is  a  prima¬ 


ry  source  of  instability.  Striking  effects  of 
instabilities  on  the  performance  of  a  com¬ 
bat  model  are  found  in  computer  arithmetic 
that  cause  global  divergences  of  simulation 
runs,  branchings  on  thresholds  set  by  deci¬ 
sion  tables  that  can  be  triggered  inadver¬ 
tently  by  small  errors  in  the  computation  of 
state  variables,  and  structurally  unstable 
decision  logic. 

Difficulties  in  reducing  and  eliminating 
structure  instability  and  structural  variance 
arise  from  the  use  of  modeling  paradigms. 
Several  paradigms  are  distinguished:  dis¬ 
crete  event  simulations,  embedded  dynam¬ 
ical  systems,  and  models  of  computer 


arithmetic.  A  dynamical  systems  view¬ 
point  illuminates  properties  of  a  model 
and  its  computer  simulation  that  are 
essential  for  verification  and  validation. 
By  viewing  computational  processes  as 
dynamical  systems  that  are  embedded  in 
discrete  event  simulation,  fundamental 
difficulties  for  models  of  the  real  world 
are  brought  to  light. 

This  report  discusses  examples  of 
unstable  behavior,  demonstrates  ways  to 
reduce  or  eliminate  sources  of  instabili¬ 
ty,  and  suggests  strategies  for  designing 
valid  combat  models  that  are  consistent, 
robust,  and  stable. 


VALIDITY  ISSUE 

(continued from  p.  9) 

could  be  tested  as  follows.  First,  the 
model’s  least-squares  parameters  would  be 
estimated  from  the  data  (analogous  to  esti¬ 
mating  regression  weights,  except  that  here 
all  parameters  are  simultaneously  estimat¬ 
ed  from  the  data).  This  can  be  done  using  a 
least-squares  program  such  as  Chandler^^. 
These  estimates  are  then  computed  in  the 
model  to  produce  a  predicted  point  to  coin¬ 
cide  with  each  data  point  shown  in  Figure 
1.  For  Equation  1,  the  resulting  predicted 
curves  would  be  parallel  in  each  panel, 
indicating  the  independence  among  factors 
predicted  by  this  model.  Since  the  data  are 
clearly  interactive,  the  expected  utility 
model  and  its  best-fit  parameter  values 
would  be  rejected  as  being  representative 
of  how  SMEs  process  and  value  this  factor 
information  in  making  their  judgments.  It 
is  this  ability  to  reject  hypotheses  that  lends 
credibility  to  conclusions  about  factor 
effects  and  models  and  SMEs’  values  to 
account  for  those  effects. 

Tests  of  model  predictions  against  data 
are  more  powerful  when  data  are  graphed 
as  in  Figure  1,  Although  statistical  tests 
are  informative  for  assessing  significance 
of  observed  effects,  they  say  nothing  about 
what  data  look  like.  Interactions  can  take 
many  forms;  different  forms  lead  to  differ¬ 
ent  conclusions  about  tradeoffs  among  fac¬ 
tors,  algebraic  models,  and  SMEs’  values 
to  account  for  the  effects.  Statistical  good- 
ness-of-fit  indices  that  correlate  predicted 
with  obtained  data  points  can  be  mislead¬ 
ing  for  assessing  a  model’s  credibility, 


because  these  indices  can  be  quite  high  for 
incorrect  models  and  even  higher  for 
an  incorrect  than  a  correct  model.  An 
incorrect  model  would  lead  to  incorrect 
conclusions  about  SMEs’  values  and  trade¬ 
offs.  Graphic  displays  (such  as  shown  in 
Figure  1)  of  predicted  and  obtained  data 
allow  assessment  of  location,  magnitude, 
and  direction  of  model/data  discrepancies 
and  thus  guidance  for  selecting  a  model 
that  could  explain  the  data. 

Once  a  model  has  received  empirical 
support  for  its  validity,  it  can  be  used  to 
predict  events  along  the  entire  continuum 
for  factors  that  have  physical  values  such 
as  Timeliness  and  Coverage.  This  is  possi¬ 
ble  because  the  function  relating  the  values 
used  in  the  experiment  to  the  model’s  scale 
values  (S  values  in  Equation  1)  can  be  esti¬ 
mated  (the  resulting  functional  relationship 
is  referred  to  as  the  Psychophysical  Func¬ 
tion  in  Psychology).  Model  values  along 
the  continuum  can  be  estimated  from  this 
function  and  computed  in  the  model  to  get 
the  model’s  predictions. 

The  importance  of  a  good  experimental- 
design/model  match  for  model  testing  can 
be  pointed  out  with  respect  to  the  factorial 
design  used  in  this  example  and  Equation 
1.  Had  the  model  in  Equation  1  been  rep¬ 
resented  as  a  simple  additive  model  with 
no  weights,  Ryj.  =  ra  (Sj  +  Sj  +  Sj^)  +  n, 
the  marginal  means  of  the  factorial  design 
would  have  been  the  least-squares  values 
of  the  factor  levels  (the  S  values)^ 
However,  for  a  model  that  multiplies 
weights  and  scale  values  as  in  Equation  1, 
a  factorial  design  as  that  illustrated  in  Fig¬ 
ure  1  does  not  permit  separate  estimates  of 


these  two  parameters;  they  are  confounded 
in  the  marginal  means.  Additional  experi¬ 
mental  design  features  are  required  to 
unconfound  these  two  parameters  so  that 
weights  and  scale  values  can  be  uniquely 
estimated.  (See  the  algebraic  logic  behind 
this  in  Veit  etal.^^.) 

For  the  data  in  Figure  1,  a  multiplicative 
model  of  the  form  Rjjj^  =  m  (SjSjSj^)  +  n, 
may  be  a  rewarding  model  to  test  since  it 
predicts  the  observed  divergent  interactions 
and  changes  in  interactions  shown  from 
Panel  A  to  Panel  C.  (See  Veit  and 
Callero^^  for  a  description  of  the  models 
that  had  to  be  rejected  for  a  similar  task, 
and  a  description  of  the  geometric  averag¬ 
ing  model  that  explained  features  of  the 
judgment  data  other  models,  including  the 
expected  utility  and  multiplicative  theories, 
failed  to  predict.) 

It  can  be  seen  that,  as  questions  are 
omitted  (i.e.,  data  points  in  Figure  1  are 
erased),  the  structure  imposed  on  the  data 
by  the  factorial  design  begins  to  disinte¬ 
grate,  as  does  one’s  ability  to  assess  the 
effects  of  factors  on  judgments  or  to  deter¬ 
mine  a  model  to  explain  those  effects. 

External  Validity 

The  second  validity  issue  is  referred  to 
here  as  external  validity  because  it  con¬ 
cerns  the  relationship  between  SME’s  val¬ 
ues  and  thoughts  to  external  “real-world” 
events.  This  validity  question  can  be  stated 
as  follows:  “Do  SMEs  accurately  perceive 
what  their  or  others’  decisions  or  behaviors 
would  be  in  specified  situations?” 

(See  VALIDITY  ISSUE,;?.  34) 
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When  using  judgments,  external  validi¬ 
ty  can  be  thought  of  as  a  second  order  level 
of  validity.  Only  when  it  is  understood 
what  and  how  factors  impact  judgments  is 
it  possible  to  explore  the  match  between 
those  judgments  and  analogous  “real 
world”  outcomes  by  manipulating  the  fac¬ 
tors  known  to  affect  SME’s  judgments  in 
the  defined  “real  world”  environment. 
(The  term  “real  world”  is  placed  in  quotes 
because  the  validity  research  medium 
selected  for  the  external  validation  test 
must  permit  manipulation  of  the  hypothe¬ 
sized  factors;  the  real  world  naturally  con¬ 
founds  them,  thus  precluding  tests  of  their 
causal  effects.)  Some  in  the  military  com¬ 
munity  have  understandably  voiced  a  great 
deal  of  concern  over  the  external  validity 
issue,  but  few  to  no  satisfactory  investiga¬ 
tions  of  external  validity  have  been  forth¬ 
coming. 

An  external  validity  test  of  the  data 
shown  in  Figure  1  could  proceed  as  fol¬ 
lows.  Let  us  suppose  that  a  field  exercise 
was  selected  as  the  external  validity  medi¬ 
um  for  this  test.  The  three  factors — Cover¬ 
age,  Timeliness,  and  Precision — would  be 
manipulated  in  the  same  4x3x3  factorial 
design  displayed  by  the  data  points  in  Fig¬ 
ure  1.  Manipulation  requires  that  factor 
levels  are  made  to  occur  and  not  occur. 
Thus,  conditions  would  have  to  be  set  so 
that  the  level  of  precision  resulted  in  a 
detection,  classification,  or  recognition  of 
the  enemy’s  vehicles  as  specified  by  the 
particular  cell  in  the  factorial  design  (or  a 
surrogate  would  have  to  be  used);  this  is 
also  the  case  for  Coverage  and  Timeliness. 
If  all  36  conditions  were  not  practically 
possible  to  perform  (perhaps  because  it 
would  be  prohibitively  expensive),  the  end 
points  of  each  factor  could  be  selected, 
thus  reducing  the  number  of  trail  cells  to  8. 
Details  of  the  experiment  would  have  to  be 
worked  out  so  that  learning  problems  that 
occur  using  people  in  exercises  is  avoided. 

In  any  case,  when  data  on  the  percent  of 
enemy  force  elements  identified  are  col¬ 
lected  under  each  different  condition  com¬ 
prising  the  experimental  trials,  these  data 
can  be  compared  in  structure  and  magni¬ 
tude  with  the  judgments.  A  superimposed 
plot  of  the  exercise  data  on  the  same  graph 
as  the  judgment  data  will  reveal  the  loca¬ 
tion,  magnitude  and  direction  of 
exercise/judgment  deviations.  These  kinds 


of  validation  methods  are  illustrated  in 
Chapter  4  of  Callero,  et  al.^^,  for  validat¬ 
ing  results  from  a  distributed  interactive 
simulation. 

The  economic  burden  of  using  live 
exercises  as  an  external  validity  medium  is 
perhaps  their  greatest  drawback.  And, 
despite  the  “realistic”  setting,  people 
express  concern  about  the  artificial  envi¬ 
ronment  of  field  experiments  as  they  do 
with  judgment  experiments.  In  field  exper¬ 
iments,  participants  do  not  experience  real 
combat.  Rather,  they  pretend  to  fight  an 
enemy  on  all  too  familiar  terrain  (e.g.,  the 
National  Training  Center)  in  a  relatively 
benign  environment.  A  further  concern  is 
the  inability  to  investigate  the  effects  of 
proposed  capabilities  on  performance  of 
systems  not  yet  fielded;  restrictions  are  to 
systems  that  exist  and  can  be  procured  for 
the  experiment.  Primarily  however, 
because  of  the  tremendous  economic  bur¬ 
den  of  field  experiments,  researchers  turn 
to  other  research  mediums  to  address  their 
research  questions.  Primary  among  those 
research  mediums  are  simulations  and 
SMEs’  judgments.  Each  medium  has  its 
associated  validity  issues. 

Comments 

If  there  is  to  be  a  military  science,  asser¬ 
tion-based  research  (i.e.,  opinions  and 
claims  in  the  absence  of  supporting  data) 
and  descriptive  research  need  to  be  placed 
in  their  proper  perspective — as  information 
bases  for  hypothesis  formulation. 
Research  methods  to  test  hypotheses  need 
to  be  adopted,  regardless  of  the  research 
medium — ^judgments,  simulations,  live 
exercises.  The  well-known  research  guide¬ 
line  for  addressing  causal  questions  in  any 
research  area,  whether  it  be  medicine,  edu¬ 
cation,  physics,  astronomy,  biology,  and  so 
on,  is  to  control  for  variables  that  could 
provide  alternative  explanations  of 
observed  results.  Experimental  control  is 
especially  important  when  policy  implica¬ 
tions  resulting  from  research  endeavors  are 
costly  —  either  in  terms  of  money  or  lives. 

A  major  purpose  of  this  article  is  to  sug¬ 
gest  to  researchers  who  choose  military 
judgment  as  a  basis  for  their  research,  to 
use  research  methods  that  place  claims  or 
policy  implications  emanating  from  judg¬ 
ment  data  in  a  framework  that  allows  them 
to  undergo  validation  tests,  that  is,  allows 
them  to  be  rejected  when  wrong.  If  SMEs’ 
judgments  are  to  be  used,  it  is  important  to 


be  able  to  appropriately  interpret  their 

meaning. 
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VEEP’S  PEEP 

(continued from  p.  5) 

Reengineering  Committee 

We  have  established  an  ad  hoc  commit¬ 
tee  this  year  to  review  the  current  structure 
of  MORS  Working  Groups  and  Composite 
Groups.  Since  the  last  comprehensive 
review  of  the  MORS  WG/CG  structure  in 
1993,  incremental  changes  have  since  been 
suggested  and  adopted.  However,  much 
has  changed  in  the  defense  and  national 
security  community  over  the  last  few 
years,  and  these  changes  may  affect  the 
types  of  operations  research  analyses  need¬ 
ed  to  support  decision-making.  Therefore, 
over  the  next  months  we  will  be  reviewing 
the  functional  structure  that  organizes  our 
members  and  our  annual  symposium.  The 
Chair  of  this  committee.  Sue  Iwanski,  has 


firmly  taken  hold  of  the  objectives  to 
review  and  either  validate  current  WG/CG 
structure  with  minor  adjustments  or  to  pre¬ 
pare  alternative  structures.  According  to 
the  committee’s  plan  of  action  and  list  of 
milestones,  during  September  she  is  com¬ 
piling  comments  from  the  membership  to 
assist  in  the  review  of  the  current  structure. 
If  you  have  suggestions  or  ideas,  please 
contact  Sue. 

In  this  review  of  recent  and  planned 
activities  for  the  meeting  operations  for 
MORS,  I  hope  you  agree  that  we  are  dedi¬ 
cated  to  enhancing  the  quality  and  useful¬ 
ness  of  military  operations  research.  We 
are  looking  forward  to  a  challenging  and 
rewarding  year!  We  appreciate  your  sug¬ 
gestions  and  involvement  -  the  following 
names,  numbers  and  e-mail  addresses  are 
provided  for  easy  reference.  O 


VP(MO) 

Kerry  Kelley,  USSTRATCOM/J502 
kelleyk@j5.stratcom.af.niil 

(402)  294-8117 

65*mORSS 

Dr.  Harry  Thie,  RAND 
thie@rand.org 

(202)296-5000  x5379 

WG/CG  Coordinator 

Jay  Wilmeth,  SETA,  Inc. 
j  wilmeth  @  seta.com 

(703)  695-1763 

Special  Meetings 

Dr.  Stuart  Starr,  MITRE 
starr@mitre.org 

(703)  883-5494 

Reengineering 

Sue  Iwanski,  Northrop  Grumman 
iwanski  @  grumman.com 

(516)346-9138 
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RADM  Pierce  Johnson 
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An  Abundance  of  Articles 

Dr,  Julian  Palmore 


The  PHALANX  queue  has  been 
growing  steadily  all  year.  After 
the  exciting  64th  Military  Opera¬ 
tions  Research  Society  Symposium  at 
Fort  Leavenworth  the  queue  grew  by  an 
additional  10  articles  to  over  two  dozen. 
What  this  means  for  everyone,  especially 
department  editors  and  authors,  is  PHA¬ 
LANX  is  well-recognized  as  a  good  place 
to  publish  to  reach  the  Military  Opera¬ 
tions  Research  community.  With  the 
abundance  of  articles  submitted  for  con¬ 
sideration  for  publication  come 
inevitable  challenges  and  opportunities. 
The  main  challenge  is  to  publish  articles 
of  interest  to  the  PHALANX  audience 


that  are  important  and  timely.  As  a  bul¬ 
letin,  PHALANX  must  be  immediately 
useful  to  its  readers.  In  each  issue  of  36 
pages,  there  is  room  for  12-18  pages  of 
articles.  An  article’s  average  length  is 
about  3-4  pages.  This  means  there  may 
be  as  few  as  four  or  as  many  as  six  arti¬ 
cles  in  an  issue.  The  PHALANX  Depart¬ 
ment  structure  has  been  rejuvenated  with 
a  total  of  twelve  departments,  fully  func¬ 
tioning.  Submitted  articles  may  appear 
either  within  a  department  or  stand 
alone.  The  implication  of  a  large  queue 
of  long  articles  is  there  will  be  fewer 
appearing  in  each  issue  than  with  shorter 
articles.  So  please  consider  the  length  of 


an  article  as  you  prepare  it  for  submis¬ 
sion  to  PHALANX.  In  our  case,  “shorter 
is  better.”  O 


Check  Out  MORS’ 
New  Home  Page! 

http://www.msosa. 
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